Package: norman-ai
The NORMAN SystemThe Novel Object Recognition Mouse Analysis Network (NORMAN) system is a package for automation of the novel object recognition behavioural assay in neuroscience studies.PyPi link:https://pypi.org/project/norman-ai/This is part of work that will be presented as a poster at theBNA Festival 2021which runs from the 12-15th of April. It was developed as part of an MSci degree at the University of Dundee. The package and repository will be updated with video outlines and other details shortly, in the run up to the BNA festival.Author: Oluwaseyi JesusanmiTwitter: @neuroseyienceContentsSystem OverviewSystem RequirementsInstallation guideVideo to DI walkthroughFunction detailsUsing Deeplabcut with NORMANGlossarySystem OverviewThe novel object recognition test (NORT) compares the time a mouse spends with a new/novel object and a familiar object in order to assess a mouses memory. The quantitative output of the test is reffered to as the discrimination index (DI), and is what many behavioural neuroscience studies use. It is calculated using (tN - tF)/(tN + tF), where tN is time spent with novel object and tF is time spent with familiar object.The NORMAN system can automatically extract the DI and other metrics from a video of a NORT. The general steps in functioning of the system are as follows:Use a deeplabcut network to track a mouse's position across a NORT video.Automatically find object locations/spatial information.Calculate relative positions of mouse to each object for every video frame.Use an artificial neural network that I built and trained to understand mouse-object interaction (NORMAN) to label each frame with the relevant interaction.Extract metrics, and visualise NORMAN's labelling.The following guide usesAnacondaand conda for environment mangement, with python for programming. The program used for environment management was anaconda prompt, for quick python access I used ipython within anaconda prompt, and for in depth script-writing and development I used spyder 4. Other programs are available.The guide assumes a very basic familiarity with anaconda/conda and python, but a user interface is available for those not comfortable coding. If help is needed feel free to contact me, I'd be happy discuss, make changes, and/or post more tutorials. I have also included a glossary in this document for some terms used.System requirements and recommendationsOperating System- Windows 10 or 11. Note: All the packages used in this project have versions avaialable for Linux and Mac, so versions supporting these operating systems could be considered with sufficient interest.GPU- For sufficient processing speed you must have a GPU. You would require an Nvidia GPU that is CUDA compatible and has at least 8Gbs of VRAM to work reliably well on both the Deeplabcut and NORMAN parts of the process.RAM- Recommend 16GbsCPU- Any modern CPU multicore CP.Tested on Windows 10 laptop i7-9750h, 16Gbs of Ram, Nvidia RTX 2060 mobile with 6Gbs of VRAM. On this hardware, I tested 5 minute novel object recognition video at 30fps and 312x480 resolution. The deeplabcut tracking step took ~2 minutes 15 seconds, and the NORMAN system analysis step took ~10 seconds. Also tested with Windows 11 laptop Ryzen 9 6900HX, 16Gbs Ram, Nvidia RTX 3070ti mobile with 8Gbs Ram.Note on previous compatibility:
The NORMAN system previously usedTensorflow-directmlmachine learning API. Tensorflow-directml has simple installation, and allows tensorflow use with hardware accelleration on ANY Directx12 capable GPU on a computer runnning Windows. That includes AMD, Nvidia, and even Intel GPUs. However it was still in active development, and at the time of writing it does not have a Tensorflow 2.X version currently available. So it is not compatible with the current versions of Deeplabcut that I reccommend to use with the NORMAN system.Installation guideThis guide assumes working knowledge of the conda/anaconda environment management system.
By the end of this section you should have an environment that works with deeplabcut, NORMAN, with GPU hardware accelleration through tensorflow-GPU utilising Nvidia CUDA. You can manage conda environments using the click-through user interface in Anaconda Navigator, or you can use the command line in anaconda prompt. This guide I will be using the anaconda prompt command line interface. For conda environment management commands, refer to thisconda cheat sheet.Firstly install Nvidia CUDA 11.4 on your system from the official websitehttps://developer.nvidia.com/cuda-toolkit-archive. Many other versions may work but 11.4 has beeen tested. After this open anaconda prompt and enter the following commands.#Create initial environment with python and tensorflow installed# norman_dlc_cuda can be any name you choosecondacreate-nnorman_dlc_cudapython=3.8tensorflow-gpu=2.5# Activate the environment so you can install more packages on itcondaactivatenorman_dlc_cuda# install the animal tracking librarypipinstall"deeplabcut[gui]"==2.3.0#install normanpipinstallnorman-ai#install required version of numpycondainstallnumpy==1.23.4Now that installation is complete, perform tests to see if installation was successful using ipython. Ipython is a python program useful for working at the command line.#open the ipython programipython#check if your graphics card is recognised by tensorflow after installationimporttensorflowastfprint("Num GPUs Available: ",len(tf.config.list_physical_devices("GPU")))#check if norman was installedimportnorman_ai.norman_functionsasnfnf.run_gui()#a user interface should appearVideo to DI walkthroughThis section will take you through the basic steps required to get a DI from a video and to visualise NORMAN'S labelling, both using the provided user interface and with code. Make sure you are in the conda environment with norman and deeplabcut installed before proceeding. Please note that the user interface is primarily for demonstration purposes. To fully take advantage of NORMAN and process in batches for your own application, the code method is what you should use.User interface methodThis uses the video file, a premade deeplabcut tracking file and an option norman model file in the demo folder of the project repository.#open the ipython programipython#import normanimportnorman_ai.norman_functionsasnf#open graphical user interface (gui)nf.run_gui()Input the video file "test_79.mp4" by clicking on "please select a video", a file selection window will open.Select the corresponding deeplabcut tracking csv "test_79_poses_filtered.csv" file by clicking on "please select a pose file".Enter which object is the video is novel to the mouse.Click accept and run video analysis. (Selecting the norman model is optional).Results will be in the bottom left.To show whether the find_objects function worked correctly, press the "Display object detection" button.Press the "Visualisation of NORMAN labelling" button to produce a video showing how NORMAN labelled the video. The video will be in the same directory as the original video, with "norman" added to the name.Play the video and see how NORMAN does.Code-based methodTo follow the code walkthrough set the working directory to the demo folder in the project repository.#import norman functionsimportnorman_ai.norman_functionsasnf#import deeplabcut functionsimportdeeplabcutasdlc#get video namevideo="test_79.mp4"Deeplabcut needs a path to a deeplabcut project config file in order for its functions to work. A pretrained network is provided in the demo folder. Please note if using a full path to the config file, it may be different to the paths in the tutorial depending on where you store the repository.#get the deeplabcut project config file location.#the r makes the string be treated as a raw stringpath_cfg=r"\nort_demo-Jesusanmi-2019-09-26\config.yaml"#analyse the video and save the results to a csvdlc.analyze_videos(path_cfg,video,videotype="mp4",save_as_csv=(True))#filter the predcitions from video analysis for smoother trackingdlc.filterpredictions(path_cfg,video,save_as_csv=(True))From here NORMAN functions use information from the video file combined with the csv file for analysis.#get path to pose documentposes=r"test_79DLC_resnet_50_nort_demoSep26shuffle1_50000_filtered.csv"#create norkid python object# stating the novel object is the left object in the experiment, if the right object is novel state "right"x=nf.norkid(video,poses,"left")#show the discrimination index, the time spent with left object, time spent with right objectprint("DI:"+str(x.di)+", Time left:"+str(x.tl)+", Time right:"+str(x.tr))#make an video visualisation of norman labelling the mouse video#this is optional as not every video result will need visualisationx.draw_vid()If you have trouble with this demo, please see the function details, which contain code examples and guidance on other functions in the NORMAN package.When using NORMAN with your own data, the most common problem is related to the find_objects() function, as not all NORT mouse chambers are compatible with the object detection algorithm. If this is the case you can use a conditional argument to draw the location of the experimental objects, and all the other parts of analysis will still be done automatically.Hold the "r" key to reset the drawing and hold the escape key to exit once the drawing is complete.#if you wish to mark object locations yourself use draw_objects as true#hold r to reset the drawing#press escape to exit the window once the drawing is completex=nf.norkid(video,poses,"left",draw_obj=True)#show the discrimination index, the time spent with left object, time spent with right objectprint("DI:"+str(x.di)+", Time left:"+str(x.tl)+", Time right:"+str(x.tr))#make an video visualisation of norman labelling the mouse video#this is optional as not every video result will need visualisationx.draw_vid()Function detailsThough very few NORMAN functions are needed to extract a discrimination index from a video, the NORMAN system has more functions for working with NORT images and videos. In this section I will explain how to use various functions in the library, with their use-cases and code examples. Where relevant I will also include information on how/why I made the function for context. Code examples below were done in the “demo” folder of the NORMAN project repository, using the “test_79.mp4” video as the source data. The top of each code example with show the arguments of the function and their defaults when applicable. For the full code of each function see the norman_functions.py file, and for the help doc strings use the python default function help().#norman help exampleimportnorman_ai.norman_functionsasnf# ask for documentation for the functionhelp(nf.find_objects)Norkid python class, object parameters and methodsThe "norkid" class handles all the processes to analyse a video, while holding the data for a video in one place. To create a norkid object you need a path to a video, a tracking csv, and which experiment object is the novel object (left or right). A norkid object stores: an image with the mouse filtered out, the find object output, the relative position function output, the labels produced by the NORMAN model, the DI and related metrics. These properties can be used in conjunction with other functions and libraries, while easing the debugging process.#%% norkid object exampleimportnorman_ai.norman_functionsasnffrommatplotlibimportpyplotasplt#get video name, norman model file and deeplabcut tracking filevideo="test_79.mp4"model_path="norman_model1.h5"pose_file="test_79_poses_filtered.csv"#make a norkid objectx=nf.norkid(video,pose_file,model_path,"left")# view name of input videox.video_name#show the median filtered imageplt.imshow(x.median_img)# show find objects outline imageplt.imshow(x.fo_img)#view object location datafranex.object_locs#view relative position dataframex.relative_pos#view norman labels per framex.labels#show the DI, seconds spent with left and right objects, and the video fpsprint(round(x.di,2),round(x.tl,2),round(x.tr,2),round(x.fps,2))-0.3620.7543.8318.07# make a norman labelled video with a default name (video name + _norman)x.draw_vid()# make a norman labelled video with a specified namex.draw_vid(out_name="my_norman_vid.mp4")Frame extractor – ext_frame()Extract a frame or multiple frames from a video. Outputs an image from a video as an image array, a file, or a folder image files. Often when working with video data, specific frames may need to be checked or analysed, as many functions work on a frame-by-frame basis.# Function arguments = ext_frame(vid_name, frame_index, out_name=False)# Extract frame exampleimportnorman_ai.norman_functionsasnf# set video namemy_video=“test_79.mp4”# Extract a single frame to produce an image arrayimage_array=nf.ext_frame(my_video,42)# Extract a frame and save as an image filenf.ext_frame(my_video,42,out_name="my_frame.png")# Extract frames 42 to 150 to the current working directorynf.ext_frame(my_video,(42,150))# Extract frames 42 to 150 to a new foldernf.ext_frame(my_video,(42,150),out_name="many_frames")Resolution changer – reso_change()Changes the resolution of a video to a specified value on the x and y axis. Creates a new video, does not affect the data of the original video.# Function Arguments: reso_change(input_vid, out_vid, res=(640, 480))# Resolution change example.#Please note new video is created, original video is not affectedimportnorman_ai.norman_functionsasnf# set video namemy_video=“test_79.mp4”#change resolution of video to default value of 640x480nf.reso_change(my_video,"new_res.mp4")#change resolution of video to a new arbitrary valuenf.reso_change(my_video,"new_res.mp4",res=(789,456))Moving object remover/filter – median_filt_video()This function removes the mouse from the video and produces an image of the bare experiment box and objects. This must be done before using the find_objects() function for object detection, as the mouse will interfere with the object finder. It will remove any moving object from a video given enough frames.#Function arguments: median_filt_video(video_name, show=False, out_file=False, select_no=15)#%% Median filter exampleimportnorman_ai.norman_functionsasnffrommatplotlibimportpyplotasplt# set video namemy_video=“test_79.mp4”#show what a single frame looks like before filteringplt.imshow(nf.ext_frame(my_video,42))#use median filter with default 15 frames, show the result, output to variable and to filex=nf.median_filt_video(my_video,show=True,out_file="filtered.png")#use median filter with specified number of frames, output to variablex=nf.median_filt_video(my_video,select_no=20)Object localiser – find_objects()This automatically finds the 2 experiment objects and outputs the locations, area and outline of each of them. The values are used later to calculate proximity of the mouse to objects. This function should be used after the median filter has been used to create a mouse-free image. So far in testing it reliably finds the location of objects on videos where the floor of the test chamber is plain. An alternative version of the function is under exploration for boxes with patterned floors or large amounts of debris, as this affects the edge detection algorithms.# Function Arguments: find_objects(image_or_path, show=False, img_out=False, im_write=False)# Find objects exampleimportnorman_ai.norman_functionsasnffrommatplotlibimportpyplotasplt# set video namemy_video=“test_79.mp4”#use median filter to generate a mouse-free image array or filebox_image=nf.median_filt_video(my_video)nf.median_filt_video(my_video,out_file="box_img.png")#show image using matplotlib. Note it may show in false colourplt.imshow(box_image)#use the object detector on an image array, show the result, save the output as pandas dataframex=nf.find_objects(box_image,show=True)#use object detector on an image file, show the result, save output as pandas dataframex=nf.find_objects("box_img.png",show=True)#use object detector and output resulting image array with dataframex,image=nf.find_objects(box_image,img_out=True)#use object detector and write the output image as a filex=nf.find_objects(box_image,im_write="outlines.png")Video maker from multiple images – make_video()This function takes in a directory of images, and outputs a video created from these images at any specified framerate.#Function arguments: make_video(image_folder, new_vid, fps=25)# Make video exampleimportnorman_ai.norman_functionsasnf# make a new video from a folder of images with an fps of 25nf.make_video("many_frames","vid_new.mp4")# make a new video from a folder of images with any fpsnf.make_video("many_frames","vid_new.mp4",fps=60)Making predictions from position data – label_vid()Prediction in the case of the NORMAN system refers to the NORMAN network model taking relative positional information per-frame, then determining what the mouse is paying attention to in each frame. In the output array, neither=0, left object =1, right object=2. These labels are used for calculating the DI or other useful metrics. Since the labels are per-frame, many NORT attention metrics can be calculated, including latency to first interaction and change in interaction amount over the time course of an experiment.#Function arguments: label_vid(model_path, relative_pos)# Label vid exampleimportnorman_ai.norman_functionsasnf#input video namevideo="test_79.mp4"#input path to pose filepose_file="test_79_poses_filtered.csv"#extract the median imagemedian_img=nf.median_filt_video(video)# extract object locationsobject_locs=nf.find_objects(median_img)# calculate relative position of mouse to objectsrelative_pos=nf.rel_pos(pose_file,object_locs)#get labels produced by the prediction by normanlabels=nf.label_vid(relative_pos)Making a NORMAN labelled video – draw_vid()The draw_vid() function visualises the NORMAN model predictions (see figure). This is useful for assessing whether the system is working correctly as it can be judged by eye. It is also useful when presenting data to show how the NORMAN system works.# Function arguments: draw_vid(y_labels, input_vid, out_name=False)# Draw vid exampleimportnorman_ai.norman_functionsasnf#input video namevideo="test_79.mp4"#input path to norman modelmodel_path="norman_model1.h5"#input path to pose filepose_file="test_79_poses_filtered.csv"#extract the median imagemedian_img=nf.median_filt_video(video)# extract object locationsobject_locs=nf.find_objects(median_img)# calculate relative position of mouse to objectsrelative_pos=nf.rel_pos(pose_file,object_locs)#get labels produced by the prediction by normanlabels=nf.label_vid(model_path,relative_pos)#make norman labelled video, with default naming of the original video name +_norman saved to the current working directorynf.draw_vid(labels,video)#make norman labelled video, with specified name saved to specified directory.nf.draw_vid(labels,video,out_name="labelled_vid.mp4")Calculating discrimination index – calc_di()This function only produced the DI, time spent with each object and the fps.# calc di exampleimportnorman_ai.norman_functionsasnf#input video namevideo="test_79.mp4"#extract labels from videopose_file="test_79_poses_filtered.csv"median_img=nf.median_filt_video(video)object_locs=nf.find_objects(median_img)relative_pos=nf.rel_pos(pose_file,object_locs)labels=nf.label_vid(relative_pos)# calculate and store just the di of a video.# we are saying the novel object is the left onedi=nf.calc_di(video,labels,"left",ret_all=False)# store the di, time with left, time with right and fps (default)#we are saying the novel object is the right onedi,tl,tr,fps=nf.calc_di(video,labels,"right")#show results rounded to 2 decimal placesprint(round(di,2),round(tl,2),round(tr,2),round(fps,2))0.3520.8143.6618.07Converting maze project csv into one compatible with NORMANNote: This function is only applicable to one specific project. This is a function to convert Deeplabcut output that was recorded from the bottom of the mouse to one compatible with the rest of the NORMAN system. It assumes the nose, mid and tailbase of the mouse marked were marked in Deeplabcut, and uses these to infer where the ears are, then creates a new CSV based on this. This csv file can then be used as a normal pose file in the system going forwards. It would be advisable to use this function in a loop to do a lot of files at once.#get name of the file you want to convertmaze_pose_file=r"maze_dlc_output.csv"#name of the the new file you want to createnew_file_name="converted_file.csv"#convert the filenf.maze_file_convert(maze_pose_file,new_file_name)Using Deeplabcut with NORMAN.Deeplabcut is a python library for marker-less tracking of animals using artificial neural networks. It provides a variety of functions centred around streamlining the process of tracking animals. The general workflow of the library is to use a neural network that has been pretrained on animal movement, train it to recognise features on the animal you wish to track by labelling images, then use the trained network to track animal movement on a selection of videos. For in depth details please refer to the officialdeeplabcut tutorials on github. Here I will briefly discuss points that are relevant for using Deeplabcut in tandem with the NORMAN system.Installation – The tutorials recommend tensorflow-gpu installation for using Deeplabcut with hardware accelleration. But the previously discussed tensorflow-directml based environment allows a lot easier installation when using a GPU and allows you to have much more customizable environments, while remaining compatible with deeplabcut.Config file – When you start a new deeplabcut project it creates a folder with a “config.yaml” file. This config file is what all the functions use to find other files in the directory, such as the location of the trained models. The entire project folder can be moved, copied and shared. As long as the information in the config file is up do date, all the functions should work as normal.Training a deeplabcut model – When labelling frames to train Deeplabcut to track mice or rats, please ensure the labels are added in the right order with the correct text. This is to ensure the output csv from deeplabcut analysis is compatible with the NORMAN system. The list of labels is: nose, l_ear, r_ear, tailbase. L_ear refers to left ear, r_ear refers the right ear, and tailbase refers to where the tail connects to the main body of the mouse. If you are training a Deeplabcut network with a GPU that has less than 8Gbs of video memory (VRAM), you may run into memory allocation errors. To solve this you must change the session parameters to allow a procedural change in memory allocation during training . The following code implements this:config=tf.ConfigProto()config.gpu_options.allow_growth=Truesess=tf.Session(config=config)The Deeplabcut model zoo – Deeplabcut has a growing “model zoo”, where you can download pre-trained models to use for animal tracking. These are very convenient as the most time consuming step (training and evaluating the model), is eliminated. There is no pretrained mouse tracking model available yet, but when one is released, this should be used and NORMAN will be given a function to accept the csv files produced when using this model.Video resolution - The processing time/resolution trade off must also be considered. Higher resolution can mean higher accuracy as there is more detail visible, but higher resolutions can greatly increase the processing time for the deeplabcut tracking step. The point of diminishing returns from resolution differs depending on the application. For example if you are tracking the nose and ears of a mouse, a very low resolution of just 480p would provide good tracking accuracy. Whereas if you were tracking whiskers at that same resolution with the mouse equidistant from the camera, the whiskers will be indistinguishable and the accuracy would suffer tremendously.GlossaryArray – A data type for storing sequences of a variety of dimensions, usually in the numpy library. It can be used to store a simple sequence of numbers, or something more complex like an image by representing each pixel colour with a number in a table-like structure. Array processing is incredibly fast, making them useful for matrices calculations.Class – In python, a class refers to a type of object and is used as a layout on which to build an object. An object in python has properties (parameters and characteristics), and methods (types of functions) associated with it. For an imaginary example, say a pet golden retriever was a python object. It would have properties such as height, weight, breed, fur colour etc. It could have methods such as run, play fetch, eat, sleep etc. These methods could also affect some of the object properties, for example the “eat” method may increase the dog object’s weight. The class for this example would be a dog class. It would be a common set of properties and methods of which the exact values of which will be different for each new dog object that is made. Python is an object orientated programming language, meaning that many data and variable are often stored as objects/object parameters, with methods which can be applied to them. For a practical implementation of a class and object, see the norkid class explanation in section 4.Computer vision – A field of computer science programming that aims to extract information from images and video. This often works by converting images into an array of numbers which represent pixels, and using algebraic functions on these numbers to extract the necessary information. Recently much of computer vision research utilises artificial neural network techniques as they can apply incredibly complex mathematical functions to images automatically, if given enough training data.
Frame/video frame – Videos are made up of many images in sequence, each individual image is called a frame. Most standard video is shot at about 30 frames per second, meaning each second of video is made out of 30 images.GPU/integrated GPU – A graphical processing unit is a specialised computer component made for processing highly parallel mathematical problems. Originally produced for computer graphics and video-games, it was discovered that the parallel nature of the hardware is well suited to running artificial neural network operations. GPUs are often parts of graphics cards, while integrated GPUs are less powerful devices that are integrated into CPU chipsets, usually for devices without dedicated graphics cards.Library/python library – Sets of reusable code. A library often includes a series of functions that are helpful for a specific type of programming problem. For example pandas is a tabular data processing library, which includes sets of functions useful for processing tabular data.Object – See entry for “Class”.Package – Package is a collection of code similar to a library. It is referred to as a package when it is released in a distributable form that can be installed into an environment.Version control – A method of archiving previous version of a coding project, commonly the git package is used to do this. Using this method all files and all changes to files are stored. This means that if the code breaks, you can roll-back to a previous version where the code was working. It is also important for scientific reproducibility, since all changes are tracked, any mistakes or instances of data being changed in a fraudulent manner can be tracked.Virtual development environment – A partition on a computer where many libraries and packages can be installed, in a way that other aspects of the computer are not affected. This allows developers to work on multiple different projects that require different packages without running into as many compatibility errors.