Package: thingscoop
<img width=800 src="resources/preview.jpg"></img>## Thingscoop: Utility for searching and filtering videos based on their content### DescriptionThingscoop is a command-line utility for analyzing videos semantically - that means searching, filtering, and describing videos based on objects, places, and other things that appear in them.When you first run thingscoop on a video file, it uses a [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) to create an "index" of what's contained in the every second of the input by repeatedly performing image classification on a frame-by-frame basis. Once an index for a video file has been created, you can search (i.e. get the start and end times of the regions in the video matching the query) and filter (i.e. create a [supercut](https://en.wikipedia.org/wiki/Supercut) of the matching regions) the input using arbitrary queries. Thingscoop uses a very basic query language that lets you to compose queries that test for the presence or absence of labels with the logical operators `!` (not), `||` (or) and `&&` (and). For example, to search a video the presence of the sky *and* the absence of the ocean: `thingscoop search 'sky && !ocean' <file>`.Right now two models are supported by thingscoop: `vgg_imagenet` uses the architecture described in ["Very Deep Convolutional Networks for Large-Scale Image Recognition"](http://arxiv.org/abs/1409.1556) to recognize objects from the [ImageNet](http://www.image-net.org/) database, and `googlenet_places` uses the architecture described in ["Going Deeper with Convolutions"](http://arxiv.org/abs/1409.4842) to recognize settings and places from the [MIT Places](http://places.csail.mit.edu/) database. You can specify which model you'd like to use by running `thingscoop models use <model>`, where `<model>` is either `vgg_imagenet` or `googlenet_places`. More models will be added soon.Thingscoop is based on [Caffe](http://caffe.berkeleyvision.org/), an open-source deep learning framework.### Installation1. Install ffmpeg, imagemagick, and ghostscript: `brew install ffmpeg imagemagick ghostscript` (Mac OS X) or `apt-get install ffmpeg imagemagick ghostscript` (Ubuntu).1. Follow the installation instructions on the [Caffe Installation page](http://caffe.berkeleyvision.org/installation.html).2. Make sure you build the Python bindings by running `make pycaffe` (on Caffe's directory).3. Set the environment variable CAFFE_ROOT to point to Caffe's directory: `export CAFFE_ROOT=[Caffe's directory]`.4. Install thingscoop: `easy_install thingscoop` or `pip install thingscoop`.### Usage#### `thingscoop search <query> <files...>`Print the start and end times (in seconds) of the regions in `<files>` that match `<query>`. Creates an index for `<file>` using the current model if it does not exist.Example output:```$ thingscoop search violin waking_life.mp4/Users/anastasis/Downloads/waking_life.mp4 148.000000 162.000000/Users/anastasis/Downloads/waking_life.mp4 176.000000 179.000000/Users/anastasis/Downloads/waking_life.mp4 180.000000 186.000000/Users/anastasis/Downloads/waking_life.mp4 189.000000 190.000000/Users/anastasis/Downloads/waking_life.mp4 192.000000 200.000000/Users/anastasis/Downloads/waking_life.mp4 211.000000 212.000000/Users/anastasis/Downloads/waking_life.mp4 222.000000 223.000000/Users/anastasis/Downloads/waking_life.mp4 235.000000 243.000000/Users/anastasis/Downloads/waking_life.mp4 247.000000 249.000000/Users/anastasis/Downloads/waking_life.mp4 251.000000 253.000000/Users/anastasis/Downloads/waking_life.mp4 254.000000 258.000000```####`thingscoop filter <query> <files...>`Generate a video compilation of the regions in the `<files>` that match `<query>`. Creates index for `<file>` using the current model if it does not exist.Example output:<a href="https://www.youtube.com/watch?v=qe9GjrUJipY"><img width=600 src="resources/filter.png"></img></a>#### `thingscoop sort <file>`Create a compilation video showing examples for every label recognized in the video (in alphabetic order). Creates an index for `<file>` using the current model if it does not exist.Example output:<a href="https://www.youtube.com/watch?v=o0VoyJgPgJE><img width=600 src="resources/clockwork_orange.png"></img></a>#### `thingscoop describe <file>`Print every label that appears in `<file>` along with the number of times it appears. Creates an index for `<file>` using the current model if it does not exist.#### `thingscoop preview <file>`Create a window that plays the input video `<file>` while also displaying the labels the model recognizes on every frame.```$ thingscoop describe koyaanisqatsi.mp4 -m googlenet_placessky 405skyscraper 363canyon 141office_building 130highway 78lighthouse 66hospital 64desert 59shower 49volcano 45underwater 44airport_terminal 43fountain 39runway 36assembly_line 35aquarium 34fire_escape 34music_studio 32bar 28amusement_park 28stage 26wheat_field 25butchers_shop 25engine_room 24slum 20butte 20igloo 20...etc```#### `thingscoop index <file>`Create an index for `<file>` using the current model if it does not exist.#### `thingscoop models list`List all models currently available in Thingscoop.```$ thingscoop models listgooglenet_imagenet            Model described in the paper "Going Deeper with Convolutions" trained on the ImageNet databasegooglenet_places              Model described in the paper "Going Deeper with Convolutions" trained on the MIT Places databasevgg_imagenet                  16-layer model described in the paper "Return of the Devil in the Details: Delving Deep into Convolutional Nets" trained on the ImageNet database```#### `thingscoop models info <model>`Print more detailed information about `<model>`.```$ thingscoop models info googlenet_placesName: googlenet_placesDescription: Model described in the paper "Going Deeper with Convolutions" trained on the MIT Places databaseDataset: MIT Places```#### `thingscoop models freeze`List all models that have already been downloaded.```$ thingscoop models freezegooglenet_placesvgg_imagenet```#### `thingscoop models current`Print the model that is currently in use.```$ thingscoop models currentgooglenet_places```#### `thingscoop models use <model>`Set the current model to `<model>`. Downloads that model locally if it hasn't been downloaded already.#### `thingscoop models download <model>`Download the model `<model>` locally.#### `thingscoop models remove <model>`Remove the model `<model>` locally.#### `thingscoop models clear`Remove all models stored locally.#### `thingscoop labels list`Print all the labels used by the current model.```$ thingscoop labels listabacusabayaabstractionacademic gownaccessoryaccordionacornacorn squashacoustic guitaractactinic radiationactionactivityadhesive bandageadjudicatoradministrative districtadmiraladornmentadventureradvocate...```#### `thingscoop labels search <regexp>`Print all the labels supported by the current model that match the regular expression `<regexp>`.```$ thingscoop labels search instrument$beating-reed instrumentbowed stringed instrumentdouble-reed instrumentfree-reed instrumentinstrumentkeyboard instrumentmeasuring instrumentmedical instrumentmusical instrumentnavigational instrumentnegotiable instrumentoptical instrumentpercussion instrumentscientific instrumentstringed instrumentsurveying instrumentwind instrument...```### Full usage options```thingscoop - Command-line utility for searching and filtering videos based on their contentUsage:thingscoop filter <query> <files>... [-o <output_path>] [-m <model>] [-s <sr>] [-c <mc>] [--recreate-index] [--gpu-mode] [--open]thingscoop search <query> <files>... [-o <output_path>] [-m <model>] [-s <sr>] [-c <mc>] [--recreate-index] [--gpu-mode]thingscoop describe <file> [-n <words>] [-m <model>] [--recreate-index] [--gpu-mode] [-c <mc>]thingscoop index <files> [-m <model>] [-s <sr>] [-c <mc>] [-r <ocr>] [--recreate-index] [--gpu-mode]thingscoop sort <file> [-m <model>] [--gpu-mode] [--min-confidence <ct>] [--max-section-length <ms>] [-i <ignore>] [--open]thingscoop preview <file> [-m <model>] [--gpu-mode] [--min-confidence <ct>]thingscoop labels list [-m <model>]thingscoop labels search <regexp> [-m <model>]thingscoop models listthingscoop models info <model>thingscoop models freezethingscoop models currentthingscoop models use <model>thingscoop models download <model>thingscoop models remove <model>thingscoop models clearOptions:--version                       Show version.-h --help                       Show this screen.-o --output <dst>               Output file for supercut-s --sample-rate <sr>           How many frames to classify per second (default = 1)-c --min-confidence <mc>        Minimum prediction confidence required to consider a label (default depends on model)-m --model <model>              Model to use (use 'thingscoop models list' to see all available models)-n --number-of-words <words>    Number of words to describe the video with (default = 5)-t --max-section-length <ms>    Max number of seconds to show examples of a label in the sorted video (default = 5)-r --min-occurrences <ocr>      Minimum number of occurrences of a label in video required for it to be shown in the sorted video (default = 2)-i --ignore-labels <labels>     Labels to ignore when creating the sorted video video--title <title>                 Title to show at the beginning of the video (sort mode only)--gpu-mode                      Enable GPU mode--recreate-index                Recreate object index for file if it already exists--open                          Open filtered video after creating it (OS X only)```### CHANGELOG#### 0.2 (8/16/2015)* Added `sort` option for creating a video compilation of all labels appearing in a video* Now using JSON for the index files#### 0.1 (8/5/2015)* Conception### LicenseMIT