Package: contentai-metadata-flatten
metadata-flatten-extractorA method to flatten generated JSON data into timed CSV events in support
of analytic workflows within theContentAI
Platform, published as the extractordsai_metadata_flatten.   There is also apypi packageof this package published for easy incorporation in other projects.Getting StartedExecutionTestingFuture DevelopmentChangesGetting StartedThis library is used as asingle-run executable.Runtime parameters can be passed for processing that configure the
returned results and can be examined in more detail in themainscript.NOTE: Not all flattening functions will respect/obey properties
defined here.force_overwrite-(bool)- force existing files to be
overwritten (default=False)compressed-(bool)- compress output CSVs instead of raw write
(default=True, e.g. append ‘.gz’)all_frames-(bool)- for video-based events, log all instances
in box or just the center (default=False)time_offset-(int)- when merging events for an asset split intomultiple parts, time in seconds (default=0); negative numbers will
cause a truncation (skip) of events happening before the zero time
mark(added v0.7.1)time_offset_source-(str)- check for this one-line file path withnumber of seconds offset according totime_offsetrules;(added v1.4.0)verbose-(bool)- verbose input/output configuration printing
(default=False)extractor-(string)- specify one extractor to flatten,
skipping nested module import (default=all, e.g.dsai_metadata)generator-(string)- cify one generator for output,
skipping nested module import (*``=all,empty=none), e.g. ``flattened_csv)GeneratorsCSV Schema (CSV)One output of this flattening will be a set of CSV files if theflattened_csvis enabled as a generator.  One file is created for each discovered/input parser/extractor.
The standard schema for these CSV files has the following fields.time_begin= time in seconds of event starttime_end= time in seconds of end (may be equal to time_start if
instantaneous)time_event= exact time in seconds (may be equal to time_start if
instantaneous)source_event= source media for event to add granularity for
event inpact (e.g. face, video, audio, speech, image, ocr, script)tag= simple text word or phrasetag_type= descriptor for type of tag; e.g. tag=concept/label/emotion, keyword=special word,
shot=segment, transcript=text, moderation=moderation, word=text/speech word, person=face or skeleton,
phrase=long utterance, face=face emotion/properties, identity=face or speaker recognition,
scene=semantic scenes/commercials/commercial_lead, brand=product or logo mention, emotion=visual or audio sentiment/emotionscore= confidence/probabilitydetails= possible bounding box or other long-form (JSON-encoded)
detailsextractor= name of extractor for insightExample Programmatic UseWhile this library is primarily used as an extractor in ContentAI, it can
be programmatically called within another extractor to simplify incoming
data into a simple list for analysis.  Several of these examples are available
as code examples in the testing scripts.This dictionary-based call example will parse output of theazure_videoindexerand return it as a dictionary only (do not generate CSV or JSON output).fromcontentai_metadata_flatten.mainimportflattendict_result=flatten({"path_content":"content/jobs","extractor":"azure_videoindexer","generator":"","verbose":True,"path_result":".",args=[])This argument call example will parse all extractor outputs and generate a CSV.fromcontentai_metadata_flatten.mainimportflattendict_result=flatten(args=["--path_content","content/jobs/example.mp4","--generator","flattened_csv","--path_result":"content/flattened")This low-level access to a parser allows more control over which file or directory
is parsed by the library and no generators are called.  This call example is the same as
the first example except that it returns aDataFrameinstead of a dictionary and may
be slightly faster.fromcontentai_metadata_flattenimportparsersimportlogginglogger=logging.getLogger()logger.setLevel(logging.INFO)list_parser=parsers.get_by_name("azure_videoindexer")parser_instance=list_parser[0]['obj']("content/jobs",logger=logger)config_default=parser_instance.default_config()result_df=parser_instance.parse(config_default)Another low-level access to parsers for only certain tag types.  This call example allows
the parsing of only certain tag types (below only those of typeidentityandface).fromcontentai_metadata_flattenimportparsersimportlogginglogger=logging.getLogger()logger.setLevel(logging.INFO)list_parser=parsers.get_by_type(["face","identity"])forparser_objinlist_parser:parser_instance=parser_obj['obj']("content/jobs",logger=logger)config_default=parser_instance.default_config()result_df=parser_instance.parse(config_default)Return ValueThe main functionmain.py::flattennow returns a richer dictionary (v1.3.0).
For programatic callers of the function the dictionary object contains adataproperty (all of the flattened data as a list) and ageneratedproperty
which contains a list of nested dictionaries indicating generated outptu (if enabled).
An example output below demonstrates the flattened results as well as two enabled generators.{'data':[{'tag':'Clock','time_begin':0,'time_end':1,'time_event':0,'score':0.08157,'details':'{"model": "/m/01x3z"}','source_event':'audio','tag_type':'tag','extractor':'example_extractor'},{'tag':'Sine wave','time_begin':0,'time_end':1,'time_event':0,'score':0.07586,'details':'{"model": "/m/01v_m0"}','source_event':'audio','tag_type':'tag','extractor':'example_extractor'},{'tag':'Tick-tock','time_begin':0,'time_end':1,'time_event':0,'score':0.07297,'details':'{"model": "/m/07qjznl"}','source_event':'audio','tag_type':'tag','extractor':'example_extractor'},...]'generated':[{'generator':'flattened_csv','path':'testme/example_extractor.csv.gz'},{'generator':'wbTimeTaggedMetadata','path':'testme/wbTimeTaggedMetadata.json.gz'}]}Execution and DeploymentThis package is meant to be run as a one-off processing tool that
aggregates the insights of other extractors.command-line standaloneRun the code as if it is an extractor. In this mode, configure a few
environment variables to let the code know where to look for content.One can also run the command-line with a single argument as input and
optionally ad runtime configuration (seeruntime
variables) as part of theEXTRACTOR_METADATAvariable as JSON.EXTRACTOR_METADATA='{"compressed":true}'Locally Run on ResultsFor utility, the above line has been wrapped in the bash scriptrun_local.sh.EXTRACTOR_METADATA='$3'EXTRACTOR_NAME=metadata-flattenEXTRACTOR_JOB_ID=1\EXTRACTOR_CONTENT_PATH=$1EXTRACTOR_CONTENT_URL=file://$1EXTRACTOR_RESULT_PATH=`pwd`/results/$2\python-umain.pyThis allows a simplified command-line specification of a run
configuration, which also allows the passage of metadata into a
configuration.Normal result generation into compressed CSVs (with overwrite)../run_local.shdata/wHaT3ver1t1sresults/Result generation with environment variables and integration of results
from a file that was split at an offset of three hours../run_local.shresults/1XMDAz9w8T1JFEKHRuNunQhRWL1/results/'{"force_overwrite":false,"time_offset":10800}'Result generation from a single extractor, with its nested directory
explicitly specified. (added v0.6.1)./run_local.shresults/dsai_metadataresults/'{"extractor":"dsai_metadata"}'Local Runs with Timing OffsetsThe scriptrun_local.shalso searches for a text file calledtiming.txtin each source directory. If found, it will offset all
results by the specified number of seconds before saving them to disk.
Also, negative numbers will cause a truncation (skip) of events
happening before the zero time mark.(added v0.7.1)This capability may be useful if you have to manually split a file into
multiple smaller files at a pre-determined time offset (e.g. three hours
-> 10800 intiming.txt).(added v0.5.2)echo"10800">1XMDAz9w8T1JFEKHRuNunQhRWL1/timing.txt./run_local.shresults/1XMDAz9w8T1JFEKHRuNunQhRWL1/results/Afterwards, new results can be added arbitrarily and the script can be
rerun in the same directory to accomodate different timing offsets.Example demonstrating integration of multiple output directories.findresults-typed-d1|xargs-I{}./run_local.sh{}results/ContentAIDeploymentDeployment is easy and follows standard ContentAI steps.contentaideploy--cpu256--memory512metadata-flattenDeploying...writingworkflow.dotdoneAlternatively, you can pass an image name to reduce rebuilding a docker
instance.dockerbuild-tmetadata-deploycontentaideploymetadata-flatten--cpu256--memory512-imetadata-deployLocally Downloading ResultsYou can locally download data from a specific job for this extractor to
directly analyze.contentaidatawHaT3ver1t1s--dirdataRun as an Extractorcontentairunhttps://bucket/video.mp4-w'digraph { aws_rekognition_video_celebs -> metadata_flatten}'JOBID:1Tfb1vPPqTQ0lVD1JDPUilB8QNrCONTENT:s3://bucket/video.mp4STATE:completeSTART:FriFeb1504:38:05PM(6minutesago)UPDATED:1minuteagoEND:FriFeb1504:43:04PM(1minuteago)DURATION:4minutesEXTRACTORSmy_extractorTASKSTATESTARTDURATION724a493complete5minutesago1minuteOr run it via the docker image…docker run --rm  -v `pwd`/:/x -e EXTRACTOR_CONTENT_PATH=/x/file.mp3 -e EXTRACTOR_RESULT_PATH=/x/result2 <docker_image>View Extractor Logs (stdout)contentailogs-f<my_extractor>my_extractorFriNov1504:39:22PMwritingsomedataJobcompletein4m58.265737799sTestingTesting is included via tox.  To launch testing for the entire package, just runtoxat the command line.
Testing can also be run for a specific file within the package by setting the evironment variableTOX_ARGS.TOX_ARG=test_basic.pytoxFuture Developmentthe remaining known extractors…openpose,dsai_tmstext_classifier_extractor,dsai_vinyl_sound_ai,dsai_name_entity_extractor,aws_rekognition_video_segmentsintegration of viewership insightscreation of sentiment and mood-based insights (which tags most
co-occur here?)ChangesA method to flatten generated JSON data into timed CSV events in support
of analytic workflows within theContentAI Platform.1.41.4.0fix for timing offsets; don’t overwrite any output if timing offset indicator1.4.1add newdsai_ads_detectorparser for predictive ad locations1.31.3.3minor fix forazure_videoindexerparsing, now first video shot cannotcontain a keyframe ?1.3.2minor fix forgcp_videointelligence_text_detectionparsing1.3.1fix for no-output generatorsfix complete output for returned dictionary of dataadd richer documentation for library/api usage1.3.0update output of main parse function to return a dict instead of file listingmodify generator specification to allow ALL (*default) or NONE for outputs1.21.2.2add parsers forgcp_videointelligence_text_detection,comskip_json,ibm_max_audio_classifier,gcp_videointelligence_object_tracking,gcp_videointelligence_people_detectionimprove testing to iterate over known set of data in testing dirfix generator/parser retrieve for whole name matches, not partialsadd documentation for new types, explicitly call outpersontag_typeupdate thedsai_activity_emotionsparser to return tag typeemotion(matching that of other AWS, Azure parsers)1.2.1updateazure_videoindexerfortag_typein detected brands (was speech, now video)1.2.0add unit-testing to package buildadd command-line / parser input as complement to contentai-driven ENV variablesfix bugs around specification of result path or specific generator1.11.1.8fix issue about constant referencefixrun_local.shscript for extra run param configfix querying for local files in non-contentai environments (regression since 1.1.0)1.1.7inclusion of other constants for compatibility with other packagesrefactor/rename of parser classes to mandate a filename output prefix (e.g.flatten_)adddsai_activity_emotionsparser (a clone ofdsai_activity_classifier)1.1.6remove applications, fork to newmetatata-databasesource, to be posted
at apypi database package1.1.4name update fordsai_moderation_imageextractor1.1.3hotfix for build distributionfix for content creation in streamlit/browsing app1.1.2deployed extractor (docker fix) for updated namespace1.1.1docs update, testing fixes, version bump for publication1.1.0rename tocontentai-metadata-flattenand publish to pypi as a package!1.01.0.2update documentation forMetadata BrowserandInventory Discoveryapp1.0.1add ability to parse input CSVs but not segment into shotmove to a single NLP library (spacy) for applications, using large model (with vectors)1.0.0add newdash/plotlydriven quality check application0.90.9.9update to optimize the pull of asset keys0.9.7upgrade to use newcontentai extractor packageupdate parser logic for safer key and data retrieval0.9.6upgrade to use newcontentai extractor packageupdate parser logic for safer key and data retrieval0.9.6small tweaks/normalization of rounding factor for extractorscorrect emotion souce type for azurerefactor app location for primary streamlit browser
- fix mode discovery for modules with specific UX interfaceupdate file listing to show data bundle files as wellrefactor utilities script for reuse in other apps0.9.5update to parse new version ofdsai_placesadd new parser fordetectron2extractor0.9.4add static file serving to streamlit app, inspired by thisstreamlit issue discussionmodify some pages to point to downloadable tables (with button click)create new download page/mode that lists the generated and source filesminor refactor of app’s docker image for better caching in local creation and testing0.9.3adddsai_moderation_textparser, updatedsai_moderationparser for version robustness
- add min threshold (0.05) to both moderation detectors0.9.2add recursion to file-based discovery method for processed assets
- unify read of JSON and text files with internalaized function call in extractor base classfix some extractors to use single name referenceself.EXTRACTOR0.9.1fix transcript parsing inazure_videoindexercomponentadd speaker differentiation as an identity block inazure_videoindexer(similar toaws_transcribe)0.9.0add timeline viewing to theevent_tablemode of streamlit app0.80.8.9fixes to main streamlit app for partial extractors (e.g. missing identity, sparse brand)0.8.8add parser fordsai_moderation0.8.7add parser fordsai_activity_classifierfix bug for faulty rejection offlatten_aws_transcriberesults0.8.6add parsers forpyscenedetect,dsai_sceneboundary,aws_transcribe,yolo3,aws_rekognition_video_text_detectadd speaker identity (from speech) togcp_videointelligence_speech_transcriptionaddtypefield (maps totag_type) to output generated bywbTimeTaggedTmetadatagenerator
- add hashing against data (e.g.box) within JSON metadata generator0.8.5add parsers fordsai_yt8m(youtube8M or mediapipe)0.8.4add parsers fordsai_activity_slowfast(activity) anddsai_places(scene/settings)addsource_typesub-field toevent_tablebrowsing mode0.8.3addmanifestoption to application for multiple assetsfix app docker file for placement/generation of code with a specific user IDfix CI/CD integration for auto launchfix app explorer bugs (derive ‘words’ from transcript/keywords if none)0.8.2hotfix for missing data indsai_metadataparser0.8.2slight refactor of how parsers are discovered, to allow search by name or type (for use as package)fix package import for contentai local fileswitchtag_typeofocrtotranscriptandocrforsource_type(azure_videoindexer)0.8.1adding music parserdsai_musicnnfor different audio regions0.8.0convert to package for other modules to installswitch document to RST from MDadd primitive testing capabilities (to be filled)0.70.7.1added truncation/trim of events before zero mark if time offset is
negativere-brand extractor asdsai_metadata_flattenfor ownership
consistency0.7.0create new set of generator class objects for varying output
generatoradd newgeneratorinput for limiting output to a single type0.60.6.2renamerekognition_face_collectiontoaws_rekognition_face_collectionfor consistency0.6.1split documentation and changesadd newcae_metadatatype of parsermodifysource_typeof detected faces inazure_videoindexertofacemodify to add newextractorinput for limit to scanning (skips
sub-dir check)0.6.0adding CI/CD script forgitlabvalidate usage as a flattening servicemodifysource_typeforaws_rekognition_video_celebstoface0.50.5.4addingface_attributesvisualization mode for exploration of face
datafix face processing to split out totag_typeasfacewith
richer subtags0.5.3add labeling component to application (for video/image inspection)fix shot duration computeation in application (do not overwrite
original event duration)add text-search for scanning named entities, words from transcript0.5.2fix bugs ingcp_videointelligence_logo_recognition(timing) andaws_rekognition_video_faces(face emotions)add new detection oftiming.txtfor integration of multiple
results and their potential time offsetsaddedverboseflag to input of main parserrenamerekognition_face_collectionfor consistency with other
parsers0.5.1split app modules into different visualization modes (overview,event_table,brand_expansion)brand_expansionuses kNN search to expand from shots with
brands to similar shots and returns those brandsevent_tableallows specific exploration of identity
(e.g. celebrities) and brands witih image/video playbackNOTEThe new application requiresscikit-learnto perform
live indexing of featuresdramatically improved frame targeting (time offset) for event
instances (video) in application0.5.0split main function into sepearate auto-discovered modulesadd new user collection detection parserrekognition_face_collection(custom face collections)0.40.4.5fixes for gcp moderation flatteningfixes for app rendering (switch most graphs to scatter plot)make all charts interactive againfix for time zone/browser challenge in rendering0.4.4fixes forazure_videoindexerparseradd sentiment and emotion summaryrework graph generation and add bran/entity search capability0.4.3add newazure_videoindexerparserswitch flattened reference fromlogotobrand;explicittomoderationadd parsing librarypytimeparsefor simpler ingestfix bug to delete old data bundle if reference files are available0.4.2add newtime_offsetparameter to environment/run configurationfix bug for reusing/rewriting existing filesadd output prefixflatten_to all generated CSVs to avoid
collision with other extractor input0.4.1fix docker image for nlp tasks, fix stop word aggregation0.4.0adding video playback (and image preview) via inline command-line
execution of ffmpeg in applicationcreate new Dockerfile.app for all-in-one explorer app creation0.30.3.2argument input capabilities for exploration appsort histograms in exploration app by count not alphabet0.3.1browsing bugfixes for exploration application0.3.0added newstreamlitcode fordata
explorer interfacebe sure to install extra packages if using this app and starting
from scratch (e.g. new flattened files)if you’re working from a cached model, you can also drop it in
from a friend0.20.2.1schema change for verb/action consistencytime_start->time_beginadd additional row fieldtag_typeto describe type of tag (seegenerated-insights)add processing typegcp_videointelligence_logo_recognitionallow compression as a requirement/input for generated files
(compressedas input)0.2.0add initial package, requirements, docker imageadd basic readme for usage exampleprocesses typesgcp_videointelligence_label,gcp_videointelligence_shot_change,gcp_videointelligence_explicit_content,gcp_videointelligence_speech_transcription,aws_rekognition_video_content_moderation,aws_rekognition_video_celebs,aws_rekognition_video_labels,aws_rekognition_video_faces,aws_rekognition_video_person_tracking,