Package: argos-toolkit
Argos: tracking multiple objects (documentation)Argos is a free, open source software toolkit to facilitate tracking
multiple objects (animals) in videos.It is distributed in two packages, this, providing tools for video
capture, annotation of training dataset, and review and correction of
tracks detected automatically by the Argos Tracking tool. The other
package is the Argos Tracking tool, distributed in the packageargos-tracker.It is described in this article:Ray, S., & Stopfer, M. A. (2021). 
Argos: A toolkit for tracking multiple animals in complex visual environments. 
Methods in Ecology and Evolution, 00, 1-11. 
https://doi.org/10.1111/2041-210X.13776Please feel free to create an issue in theissue trackerif you have questions,
suggestions, or bugs to report.There are many other excellent tools for tracking animal behavior, but Argos:Does not require constant background.Can use classical segmentation methods like thresholding, contour
detection, and watershed when objects have good contrast with
background.Can use a fast Convolutional Neural Network to detect objects in a
noisy environment, or when the objects do not have uniform color.Does not assume fixed number of animals: animals can enter and exit
a visual scene.Is a complete toolkit with tools toOnly capture or extract from videos scenes with significant
movement, reducing the amount of data to be processed.Facilitate generation of training data from images using a GUI.Track animals in videos (as described above) using either a GUI
or in batch mode.Review and correct detected tracks using a GUI.The user documentation for Argos is available onreadthedocs.Argos is in the Public Domain, while thetracker toolis released under GPL. Both are free and open source.Instead of reinventing the wheel, we put together algorithms developed
by researchers in computer vision in a simple graphical interface to
facilitate the work of biologists.The components of Argos are loosely coupled. Specifically, the
tracking does not use neural nets, so no need for training it with
videos. You need to train the neural net for detection if you want to
use it, and that can be done with about a hundred training images
(here is a jupyter notebook showing how to train Argos/YOLACT on
Google Colab:https://github.com/subhacom/argos_tutorials/blob/main/ArgosTrainingTutorial.ipynb). Moreover,
if your target animal is in the COCO database (sheep, elephant,
giraffe, zebra, horse, dog, bird), you can freely download and use
pretrained network weights.Getting startedInstallation using AnacondaInstallanacondapython distribution. You can
download the free Individual EditionhereCreate an environment with required packages (enter this commands in
Anaconda prompt):conda create -n track -c conda-forge python cython scipy numpy scikit-learn pyqt pyyaml matplotlib pandas pytables ffmpeg sortedcontainersThis will create a virtual Python environment calledtrackActivate the environment (enter this commands in Anaconda prompt):conda activate trackInstall OpenCV with contributed modules (required for some recent tracking
algorithms, but not part of the main OpenCV distribution available in conda):pip install opencv-contrib-pythonInstall PyTorch.If you have a CUDA capable GPU, seepytorch websiteto select the  right command. But note that you will need to install the appropriateNVIDIA driverfor it to work.In case you do not have a CUDA capable GPU, you have to useCPU-onlyversion (which can be ~10 times slower), in the Anaconda
promptconda install pytorch torchvision cpuonly -c pytorchInstallpycocotoolsOn Windows:InstallMS Visual Studio Build Tools.
Select Windows XX SDK for your version of Windows.Go toC:\Program Files (x86)\Microsoft Visual C++ Build Toolsand runvcbuildtools_msbuild.batInstallgitIn the Anaconda command prompt run:pip install git+https://github.com/philferriere/cocoapi.git#egg=pycocotools^&subdirectory=PythonAPIIf using PowerShell, use this instead:pip install "git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI"On Linux/Unix/Mac you need to havemakeandg++installed, and then in
the Anaconda command prompt:pip install pycocotoolsInstall Argos with this command:pip install argos_toolkitand the Tracking utility:pip install argos_trackerIf you want the development versions of these, do this instead:pip install https://github.com/subhacom/argos/archive/master.zip
pip install https://github.com/subhacom/argos_tracker/archive/master.zipUsageTo try Argos tracking on objects in COCO dataset, download the
pretrained model released with YOLACThereor go toYOLACT repositoryto find
a mirror.  The corresponding configuration file is already installed
in{your_python_environment}/lib/site-packages/argos/config/yolact_base/yolact_base_config.yml.If you used Anaconda as described here,{your_python_environment}should beC:\Users\{username}\Anaconda3\env\track\for Anaconda3 on
Windows,~/.conda/envs/trackon Linux.You can also download some weights and corresponding configuration
files here:Argos example configTo run any of the utilities in the Argos toolkit, you have to switch
to the Anaconda environment created during the installation:conda activate trackAnd then run the utility (except the Tracking tool) as a Python module:python -m argos.{utility}Continue reading below, or refer to the documentation for specific
cases.Interactive trackingTheargos_trackutility provides a graphical
interface to set the parameters, choose algorithms for detection
(instance segmentation) and tracking, and to view the performance as
tracking proceeds. Follow the steps below to start and use this
program.Runargostracking script on the Anaconda prompt:python -m argos_trackThis will start the Graphical User Interface for the
tracker. Follow the steps below in the GUI to track objects while
visualizing the tracking.Open the video file using either theFilemenu. After selecting
the video file, you will be prompted to:Select output data directory/file. You have a choice of CSV
(text) or HDF5 (binary) format. HDF5 is recommended.Select Yolact configuration file, go to theconfigdirectory inside argos directory and selectyolact_base/yolact_base_config.yml.File containing trained network weights, and here you should
select theyolact_base_54_800000.pthfile downloaded from
YOLACT repository page.Start tracking: click theplay/pausebutton and you should see
the tracked locusts. The data will be saved in the filename you
entered in step above.If you choose CSV above, the bounding boxes of the segmented
objects will be saved in{videofile}.seg.csvwith each row
containingframe-no,x,y,w,hwhere (x, y) is the coordinate of
the top left corner of the bounding box andwandhare its
width and height respectively.The tracks will be saved in{videofile}.trk.csv. Each row in this file
containsframe-no,track-id,x,y,w,h.If you choose HDF5 in step 10.1 above, the same data will be saved
in a single file compatible with the Pandas library. The
segementation data will be saved in the group/segmentedand
tracks will be saved in the group/tracked. The actual values
are in the dataset namedtableinside each group, with columns
in same order as described above for CSV file. You can load the
tracks in a Pandas data frame in python with the code fragment:tracks = pandas.read_hdf(tracked_filename, 'tracked')Classical segmentation: Using theSegmentation methodmenu you can switch
from YOLACT to classical image segmentation for detecting target objects.
This method uses patterns in the pixel values in the image to detect
contiguous patches. If your target objects are small but have high contrast
with the background, this may give tighter bounding boxes, and thus more
accurate tracking.When this is enabled, the right panel will allow you to set the parameters.The classical segmentation works by first blurring the image so that sharp
edges of objects are smoothed out. Blur width and SD control this.Select invert thresholding for dark objects on light background.Segmentation method: if Threshold, then contiguous white pixels are taken
as objects. If DBSCAN, the DBSCAN algorithm is used for spatially
clustering the white pixels. This usually provides finer selection of
object at the cost of speed.When DBSCAN is chosen, only clusters of at leastminimum samplespixels
are considered valid.The initial segmentation is further refined by specifying limits on object
size by specifyingminimum pixels,maximum pixels,minimum width,maximum width,minimum lengthandmaximum length.Batch trackingYou can also run the tracking in batch mode from the command
line. This is useful for processing a number of files from a shell
script. This uses YOLACT for decteting objects and SORT for tracking.python -m argos_track.batchtrack -i {input_file} -o {output_file} -c {yolact_config} -w {yolact_weights} -s {score} -k {max_objects} --hmin {minimum_height} --hmax {maximum_height} --wmin {minimum_width} --wmax {maximum_width} --overlap {minimum_overlap} --max_age {maximum_misses}where every entry inside braces is to be replaced by the appropriate
value. The arguments are described below (full list can be obtained by
the commandpython -m argos_track.batchtrack -h)input_file: path of input video fileoutput_file: path of output datayolact_config: path of yolact configuration file (as described above
in step 10)yolact_weights: path of yolact trained weights/network file (as
described above in step 10.3)score: a decimal fraction between 0 and 1 specifying acceptable detection
score. 0.15 is more lenient and 0.75 is more strict. For weights
trained to detect a single object 0.15 to 0.3 can be usable.max_objects: maximum number of object to retain. This keeps at most top k
objects with maximum detection score.minimum_height: an integer - filter out detected objects whose
bounding box has length in pixels less than this number.maximum_height: an integer - filter out detected objects whose
bounding box has length in pixels larger than this number.minimum_width: an integer - filter out detected objects whose
bounding box has width in pixels less than this number.maximum_width: an integer - filter out detected objects whose
bounding box has width in pixels larger than this number.minimum_overlap: a decimal fraction between 0 and 1 - the minimum
overlap between two overlapping objects in successive frames to
consider them the same object. This overlap is measured as the ratio
of intersection to union of their bounding boxes. Smaller value will
be lenient, larger value will be stricter.Imagine object A in frame 1 has moved in frame 2 to A'. If the area
of overlap of the bounding boxes of A and A' is half their combined
area, and the specified minimum overlap is 0.3, then A' will be
correctly labeled the same as A. If the specified minimum overlap is
0.7, then A' will be considered a different object and will receive
a new label.Thus with a larger value for overlap, a small movement may cause the
object to be labeled as a new object. A smaller value of overlap may
cause different objects coming close together to be confused as the
same object.maximum_misses: if an object cannot be detected in these many
successive frames, it is considered lost. It can be smaller when
detection is good and the video is recorded at high FPS.Example:For detecting animals that should be within 5 and 50 pixels wide and
between 10 and 100 pixels long, with the yolact configuration file inconfig/yolact_config.ymland weights of a network trained to detect
these animals inconfig/weights.pth, the recorded video inmyvideo.avi, where we know that no more than 20 animals (including
misdetection of other objects as the animal, e.g. a scratch in the
arena) should be detected in the video, the following command may
work:python -m argos_track.batchtrack -i myvideo.avi -o myvideo.h5 -c config/yolact.yml -w config/weights.pth -s 0.3 -k 20 --hmin 10 --hmax 100 --wmin 5 --wmax 50 --overlap 0.3 --max_age 20This will give a new label to an object if it is missing for 20 frames
or more. If there are misdetections, they can be corrected manually by
thereviewtool described below.Before embarking on processing a series of similar videos in batch
mode, it is a good idea to track a few of them in interactive mode
described earlier in order to estimate, by trial and error, the
command line arguments like minimum and maximum height and width,
overlap and maximum number of objects.Additional utilitiesCapture video with timestamp for each framecapture.py: a python script to record from a webcam or convert an
existing video based on movement. For very long recordings it may
be wasteful to record video when there is nothing happening. You
can use this script to record or convert video so only parts where
there is some minimum change of pixels (for example due to
movement) are stored. Alongside the output video, it keeps a.csvfile with the time of each frame. Check the source code or enterpython argos/capture.py -hto find the command line arguments.Review tracks to manually correct mislabelingsreview.py: a Python/Qt GUI to go through the automatically detected
tracks and correct mistakes.Follow steps underUsageabove after installation to prepare for
running the reviewer.Start the GUI using the commandpython -m argos.reviewFrom the File menu open the track generated byargos_trackand it
will ask for the corresponding video file.Once both are selected, you will see the current frame in the right
pane and the previous frame in the left pane (initially empty).Press Play (keyboard shortcut:space bar) to start going through the video.Some videos demonstrating how to use the Review tool in complex scenarios:https://www.youtube.com/playlist?list=PLdh_edYAAElVLiBgBnLLKZY8VflMm1-LUImportant items in Menu/ToolbarScroll views together- zooming will work simultaneously on both
left and right pane, scrolling right pane will scroll the left one
too. Useful for comparing the same regions in a zoomed in video.Set colorbutton for selecting a single color for all bounding
boxes and track label text.Autocolorbutton when checked, will automatically pick random
colors.Colormapbutton for selecting a colormap and number of different
values to use from this colormap for the bounding boxes and track
label text.Show in grayscalewill show the video in gray. Helps when the
colors of bboxes and labels are too similar to the colors in the
video.If theShow popup message for left/right mismatchbutton is
checked (default), then it will show a popup message each time the
track ids in the current frame do not match those on the left frame
and the video will pause.IfShow popup message for new trackbutton is checked, then only
when a new track appears on the right pane, the video will pause a
popup message will inform you about it.IfNo popup message for tracksbutton is checked, then the video
will run silently.Whenever there is a left-right mismatch in track labels, there will
be a message in the status bar (a status message) pointing out the
differences. In the message text, new track labels will be in bold.In case a track has been mislabeled, you can drag and drop the
correct label pressing the left mouse button from the list of all
tracks in the middle to the corresponding track id in the list of
current tracks in the right list.To apply this for just the current frame, keep theShiftkey
pressed when dragging and dropping.If a track on the right is a false detection, you can delete it by
pressingxordeletekey.To apply this for current frame only keep theCtrlkey pressed at
the same time.Sometimes the identities of two objects that are too close together
or cross each other, can be swapped by mistake. You can use the
right mouse button to drag and drop one track id from the left/all
list on another on the right list to swap them.To apply this for just the current frame, keep theShiftkey
pressed when dragging and dropping.NOTESwapping and assigning on the same trackid within a single
frame can be problematic.  Sometimes the tracking algorithm can
temporarily mislabel tracks. For example, objectA(trackid=1)
crosses over objectB(trackid=2) and after the crossover
objectAgot new label as trackid=3, and objectBgot
mislabelled as trackid=1. The best order of action here is to (a)
swap 3 and 1, and then (b) assign 2 to 3. This is because
sometimes the label ofBgets fixed automatically by the
algorithm after a couple of frames. Since the swap is applied
first,B's 3 becomes 1, but there is no 1 to be switched to 3,
thus there is no trackid 3 in the tracks list, and the assignment
does not happen, andAremains 2. Had we first done the
assignment and then the swap,Bwill get the label 2 from the
assignment first, and asAalso has label 2, both of them will
become 1 after the swap.To undo the changes made in current frame pressCtrl+Z. Note that
this will undo all operations (swap, assignment, deletion)
specified in the current frame.By default the reviewer only shows the current tracks on the right
and previous frame's tracks on the left. In order to display tracks
from past frames, check theShow old tracksbutton in the toolbar
or the item inViewmenu.You can selectView->Show list of changes(keyboardAlt+C) to
display all the delete, assign and swap operations you suggested
till the current frame. These are applied during the display of
tracks, and when you save the data fromFile->Save reviewed data,
the data will be saved after applying all these changes. You can
also save the changes in a text file. This is useful if you are
unsure of the changes you are making, and do not want to make
permanent modifications or go through relatively slow full save of
all track data. You can load the original track file later and load
the change list, and these changes will be applied when you play the
video.View->Set old track age limitwill allow you to enter the number
of past frames from which old tracks will be shown when theShow old tracksmenu item is selected. This helps avoid visual clutter,
but if too short, you will miss a track id that was lost from
detection for longer than these many frames. The optimal number will
depend on the quality of the original tracking, but starting with
something between 200 and 500 may help.Selecting a trackid in the right track list will show the positions
of this track label across the entire dataset in the right pane
using a colormap with color value gradually changing with age. (TODO
check)Selecting a trackid in the middle track list will show the positions
of this track label across the entire dataset in the left pane using
a colormap with color value gradually changing with age. (TODO
check)Note that the changes like swap, assign and delete are consolidated
only when you save the data to a file. If the tracking algorithm
lost trackid 34 in frame 1000, and relabeled it 40 in frame 1001
till frame 2000, even if you assigned 34 to 40, selecting 34 in the
middle track list (all tracks) will display its positions only up to
frame 1000 in the left pane. However, if you save the data into a
file after the assigment, and then select 34 in the all tracks list,
positions of track 34 will be displayed all the way to frame 2000.Filtering out bad tracks by ROI: if there is too much noise outside
the area of interest in the video, and the tracker misclassified
these as valid tracks, you can draw a polygon in the right pane over
the video frame (after pausing the video). Click the left mouse
button to place a vertex of the polygon, at the end click it again at
the first vertex to complete the polygon. This will be the region of
interest and any object whose bounding box is entirely outside this
region of interest will be excluded.Filtering out bad tracks by size: if your objects of interest have a
specific size range, you can filter out bad detections by setting a
size limit viaView->Size limits.plot_tracks.py: a python script with functions to display the tracks.Known-issuesI get the error message when switching to CSRT:AttributeError: module 'cv2.cv2' has no attribute 'TrackerCSRT_create'You have installed the standard opecv / python-opencv /opencv-python package.
For CSRT you need opencv with contribute modules. Try:conda remove opencv-python
pip install opencv-contrib-pythonI get this exception when running argos after installationValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObjectThis happens with older version of numpy (should be 1.20 or later).
Try upgrading numpy:pip install -U numpyWhen installing withpipI get this error messageCollecting torch
  Downloading torch-1.8.0-cp39-cp39-manylinux1_x86_64.whl (735.5 MB)

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.Try pip with--no-cache-diroption, like this:pip install --no-cache-dir torchI get this exception when trying to run argos trackerModuleNotFoundError: No module named 'pycocotools'This indicates that pycocotools is not installed on your system. We
did not include pycocotools in the dependencies as that creates
problem for MS Windows (see special case for Windows in installation
instrctions above).On Unix-like systems (Linux/Mac) you can install pycocotools withpip install pycocotoolsI get this error when tryingpython -m argos_track:RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspxCheck if you have NVIDIA drivers for CUDA installed. Also note that
CUDA does not work from Windows Subsystem for Linux (WSL). In
general it is a good idea to install Argos on the native platform.Trying to train YOLACT raises this RuntimeError:RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'.This is a known bug in pytorch as of version 1.9.0 (seehttps://github.com/pytorch/pytorch/issues/44714). Try installing the
LTS version:conda install pytorch torchvision cuda=10.2 -c pytorch-lts.For various other commands to install pytorch seehttps://pytorch.orgCreditsSORTwas developed by Alex
Bewley. The related publication is:@inproceedings{Bewley2016_sort,
  author={Bewley, Alex and Ge, Zongyuan and Ott, Lionel and Ramos, Fabio and Upcroft, Ben},
  booktitle={2016 IEEE International Conference on Image Processing (ICIP)},
  title={Simple online and realtime tracking},
  year={2016},
  pages={3464-3468},
  keywords={Benchmark testing;Complexity theory;Detectors;Kalman filters;Target tracking;Visualization;Computer Vision;Data Association;Detection;Multiple Object Tracking},
  doi={10.1109/ICIP.2016.7533003}
}YOLACTwas developed by Daniel
Bolya. The related publication is:@inproceedings{yolact-iccv2019,
author    = {Daniel Bolya and Chong Zhou and Fanyi Xiao and Yong Jae Lee},
title     = {YOLACT: {Real-time} Instance Segmentation},
booktitle = {ICCV},
year      = {2019},
}If you use Argos for a publication, please cite the article describing it:Ray, S., & Stopfer, M. A. (2021).  Argos: A toolkit for tracking
multiple animals in complex visual environments. Methods in Ecology
and Evolution, 00, 1- 11.https://doi.org/10.1111/2041-210X.13776