Package: cognite-synthetic-tags
Synthetic TagsAn easy way to retrieve values from CDF and execute mathematical operations on them at the same time.MotivationTo replace a custom-made structure like this:{"EG_SecProd_Biocid":[("EG_13FI1349L.Y",+1),("EG_13FI1350L.Y",+1),],"EG_PrimProd_Scale":[("EG_13FI1318L.Y","EG_13XV1322.Y",+1),("EG_13FI1418L.Y","EG_13XV1422.Y",+1),("EG_13FI1518L.Y","EG_13XV1522.Y",+1),("EG_13FI1618L.Y","EG_13XV1622.Y",+1),...with something more readable and powerful:{"EG_SecProd_Biocid":Tag("EG_13FI1349L.Y")+Tag("EG_13FI1350L.Y"),"EG_PrimProd_Scale":(Tag("EG_13FI1318L.Y")*Tag("EG_13XV1322.Y")+Tag("EG_13FI1418L.Y")*Tag("EG_13XV1422.Y")+Tag("EG_13FI1518L.Y")*Tag("EG_13XV1522.Y")+Tag("EG_13FI1618L.Y")*Tag("EG_13XV1622.Y")+...Traditionally (by using Cognite SDK for Python), to retrieve a moderately complex set of data from CDF, we would:make a list of all the tagsfetch from CDF and get a dataframerename columns in the dataframeperform column-wise calculationsperform any calculations specific to particular rowsWithSynthetic Tagsthis becomes:define any custom column-wise calculationsmake a dict with specifications for columns (a single tag, or an expression involving multiple tags and functions)fetch and calculate in one stepAdvantagesExpressive Syntax>>>specs={..."some_valve":Tag("FOO.123"),..."another_meter":Tag("BAR-456"),..."sum_of_2_things":Tag("THING_A")+Tag("THING_B"),...}>>>TagResolver(retrieval_function).latest(specs){"some_valve":42.0000123,"another_meter":42000.456000,"sum_of_2_things":78.9,}Tagclass is used as a reference for values that are going to be fetched from API. It "understands" many common
algebra operations such as:basic math operations:total_a_b = Tag("METER_A") + Tag("METER_B")parenthesis and literal values:complicated_calculation = (Tag("METER_C") - 10) / (Tag("METER_D") + TAG("METER_E"))"boolean logic:alert_status = Tag("METER_F") > 42It also supports function calls, either on multiple tags or on individual tags:calculations on individual tags:value_int = Tag("MEETER_G").calc(round)functions with multiple tags:value_foo = Tag.apply(bar, Tag("MEETER_H"), Tag("MEETER_I"), Tag("MEETER_J"))baris a function, seeCalculations with Multiple Tagssection below.TagResolveris where the actual call to the API happens, and whereTaginstances are replaced with actual values
and where math operations are performed. Besideslatestthere is alsoresolveanddf(find more
info of all three below).ExtendabilityEasily extendable with additional function calls and / or math operations.Calculations on a Single TagTag.calcmethod takes a callable which will be applied to the result (element-wise) after the value has been fetched
from CDF.>>>defgalons_per_minute(val):...returnval*4.40287>>>specs={..."flow_in_sm3_per_hour":Tag("FLOW_METER.123"),..."flow_in_galons_per_minute":Tag("FLOW_METER.123").calc(galons_per_minute),...}>>>TagResolver(retrieval_function).latest(specs){"flow_in_sm3_per_hour":12.3456,"flow_in_galons_per_minute":54.35604149,}Calculations with Multiple TagsTag.applyis a class method that takes a callable and any number ofTaginstances. When the values are fetched from
CDF, the callable will be applied (element-wise) with the tag values passed to it as arguments.>>>defclosest_to_42(*vals):..."""Return whichever value in `vals` is closest to 42"""...deltas=[abs(42-val)forvalinvals]...returnvals[deltas.index(min(deltas))]>>>specs={..."value_1":Tag("METER_A"),..."value_2":Tag("METER_B"),..."value_3":Tag("METER_C"),..."answer_to_everything":Tag.apply(closest_to_42,Tag("METER_A"),Tag("METER_B"),Tag("METER_C")),...}>>>TagResolver(retrieval_function).latest(specs){"value_1":11,"value_2":44,"value_3":57,"answer_to_everything":44,}Tag.calcis a ShorthandTag.calcis provided as a convenience method and for a more readable syntax.
It is equivalent toTag.applywith a single argument:# These two lines are equivalent:Tag("FLOW_METER.123").calc(galons_per_minute)Tag.apply(galons_per_minute,Tag("FLOW_METER.123"))Referencing Functions by NameBothcalcandapplyaccept a string instead of a callable for their first argument. In this case, the string
must match a key in a dict passed toTagResolver. This dict contains the actual callables which are then used as
described above.This feature can be used to address issues with importing Python modules, or to specify short functions usinglambda.>>>my_extension={..."galons_per_minute":lambdaval:returnval*4.40287,...}>>>specs={..."flow_in_sm3_per_hour":Tag("FLOW_METER.123"),..."flow_in_galons_per_minute":Tag("FLOW_METER.123").calc("galons_per_minute"),...}>>>TagResolver(retrieval_function,my_extension).latest(specs){"flow_in_sm3_per_hour":12.3456,"flow_in_galons_per_minute":54.35604149,}>>>defclosest_to_42(*vals):..."""Return whichever value in `vals` is closest to 42"""...deltas=[abs(42-val)forvalinvals]...returnvals[deltas.index(min(deltas))]>>>specs={..."value_1":Tag("METER_A"),..."value_2":Tag("METER_B"),..."value_3":Tag("METER_C"),..."answer_to_everything":Tag.apply("nearest_42",Tag("METER_A"),Tag("METER_B"),Tag("METER_C")),...}>>>TagResolver(retrieval_function,{"nearest_42":closest_to_42}).latest(specs){"value_1":11,"value_2":44,"value_3":57,"answer_to_everything":44,}Caching and Combined API CallsAny call toTagResolver.latest(ordforresolve) will result in the minimum number of calls
to the API to retrieves all needed values.Each instance ofTagResolverkeeps internal cache and only queries the API for tags that are needed.In the next example with multiple calls tolatest(), the CDF time series API endpoint is hit only once.>>>resolver=TagResolver(retrieval_function)>>>resolver.latest({..."value_1":Tag("METER_A"),..."value_2":Tag("METER_B"),..."value_3":Tag("METER_C"),..."val_1_and_2":Tag("METER_A")+Tag("METER_B"),..."val_2_and_3":Tag("METER_B")+Tag("METER_C"),..."val_1_and_3":Tag("METER_A")+Tag("METER_C"),...}){"value_1":12,"value_2":23,"value_3":34,"val_1_and_2":35,"val_2_and_3":...}>>>resolver.latest({..."value_1":Tag("METER_A"),..."value_1_percent":100*Tag("METER_A")/(Tag("METER_A")+Tag("METER_B")+Tag("METER_C")),...}){"value_1":12,"value_1_percent":17.3913043478}>>>resolver.latest({..."value_2":Tag("METER_B"),..."value_2_percent":100*Tag("METER_B")/(Tag("METER_A")+Tag("METER_B")+Tag("METER_C")),...}){"value_2":23,"value_2_percent":33.3333333333}Avoiding CacheIn case that the caching is not desired (i.e. if we wanted to query CDF again in each of the threelatest()calls above) we should create a new instance ofTagResolverfor each one (i.e. useTagResolver(retrieval_function).latestinstead ofresolver.latest).Multi-value Lookups (Series)While the primary motivation forSynthetic Tagslibrary was to facilitate single-value lookups, as seen in the
examples so far, the library also supports retrieving of multiple datapoints per tag (aspd.Series) as well as
performing element-wise operations on them.See "Full examples" section below for more examples with Series.Multiple Data StoresTagclass supports specifying a string key for an alternative data store (a.k.a. the fetch function).
The corresponding argument has to be present in theTagResolverconstructor call. This allows us to mix
and match values from tags obtained from separate CDF API calls.For example, we can fetch a single average value of a time series and then multiply it with a series of values from
another time series (or, indeed, the same one if desired):>>>resolver=TagResolver(get_series,average=get_average)>>>resolver.latest({..."avg_value":Tag("METER_A","average"),..."above_average":Tag("METER_A")>Tag("METER_A","average"),...}){"avg_value":42,"above_average: <pd.Series... >,  # series of bool values, True for points that are above 42, False for others}Note: All tags using any particular value store are gathered in a single API call, so no matter how many tags there are,
the number of calls to the CDF API will always be equal to the number of value stores.There are two caveats to this:Any value stores that don't have any tags are not used, i.e. no calls there.For really large number of tags, the CDF SDK might split up a single large query into multiple smaller queries
that it executes in parallel and then combines the results of. This is an internal implementation detail of
the Python CDF SDK and does not affect (not is being affected) byCognite Synthetic Tags.LimitationsBoolean expressionsPython does not allow overloading boolean operations, sobool(Tag(...))is not allowed (it will raise aValueErrorwith an extensive explanation.)This is because expressions likeTag(A) or Tag(B)get evaluated immediately, there is no (good) way to defer the
evaluation until values have been fetched from CDF.Use Bit-wise OperatorsExpressions likeTag(A) | Tag(B)work as expected.Full list of bit-wise operators:and:&or:|xor:^not:~Don't Use Ternary OperatorBecause of the limitations discussed here, there is no way to use ternary operator with instances ofTagclass:specs={"foo":Tag(A)ifTag(A)>0elseTag(B),# will not work!}Instead, define a custom operation and useTag.applyto apply it:defpositive_a_or_b(a,b):returnaifa>0elsebspecs={"foo":Tag.apply(positive_a_or_b,Tag(A),Tag(B)),}Comparison with Synthetic Time Series APICDF API supportsSynthetic Time Series. This library
provides similar functionality and there is a significant overlap (e.g. both can apply trigonometric functions on
CDF values).The main difference is that Synthetic Time Series performs calculations on the server, whereasSynthetic Tagsfetches only basic data from the API and performs the calculations locally.Performing the calculations serverside means less code and fewer opportunities for bugs.Performing the calculations locally means more control and extendability.In principle some support for the Synthetic Time Series API could be added to this library. TDB.Full ExamplesThis section uses data from OpenIndustrialData project.Env for these examples:COGNITE_PROJECT='publicdata'COGNITE_CLIENT_NAME='testing_synth_tags'# anythingCOGNITE_API_KEY='<API_KEY>'Get your API key fromhttps://openindustrialdata.com/get-started/Imports and Data Storage Callable>>>fromcognite_synthetic_tagsimportpoint_at_time,series,Tag,TagResolver>>>fromcognite.clientimportCogniteClient>>># CONFIGURE FETCHING PARAMS:>>>client=CogniteClient()>>>get_latest=point_at_time(...client,...query_by="external_id",...at_time="90d-ago",...lookbehind_start_time="91d-ago",...)>>>get_average=point_at_time(...client,...query_by="external_id",...at_time="90d-ago",...lookbehind_limit=10,...aggregate="average",...granularity="1h",...)>>>get_series=series(...client,...query_by="external_id",...start="91d-ago",...end="90d-ago",...aggregate="average",...granularity="1h",...)>>># just for readability:>>>VALVE_22="houston.ro.REMOTE_AI[22]">>>METER_A="houston.ro.REMOTE_AI[3]">>>METER_B="houston.ro.REMOTE_AI[4]">>>METER_C="houston.ro.REMOTE_AI[5]"Simple Usage>>># single value (not very useful):>>>tag_resolver=TagResolver(get_latest)>>>specs={"valve":Tag(VALVE_22)}>>>tag_resolver.series(specs){'valve':0.003925000131130218}>>># simple multiplication:>>>specs={"valve_percent":100*Tag(VALVE_22)}>>>tag_resolver.series(specs){'valve_percent':39.25000130113021085}>>># apply a function to a value:>>>specs={"valve_percent_int":(100*Tag(VALVE_22)).calc(round)}>>>tag_resolver.series(specs){'valve_percent_int':39}Notice in the last example we calcalcon the result of100 * Tag(...). This works because whenever  aTaginstance encounters a math operator, it combines with other operands (the liberal value100in this example) to
create a newTaginstance. We have calledcalcmethod on this newTaginstance.>>># fetch multiple values in a single API call and also perform some math:>>>specs={..."pressure_1":Tag(METER_A),..."pressure_2":Tag(METER_B),..."pressure_diff":Tag(METER_A)-Tag(METER_B),..."p1_percent":Tag(METER_A)/(Tag(METER_A)+Tag(METER_B))*100,...}>>>tag_resolver.series(specs){'pressure_1':1.4,'pressure_2':35.8,'pressure_diff':-34.4,'p1_percent':3.763440860215054}Usage with Series>>>specs={"valve":Tag(VALVE_22)}>>>tag_resolver=TagResolver(get_series)>>>tag_resolver.series(specs){'valve':2021-06-1617:21:080.2394252021-06-1617:21:091.3502002021-06-1617:21:102.7435762021-06-1617:21:113.5442762021-06-1617:21:123.8739742021-06-1617:21:144.2547002021-06-1617:21:154.5098262021-06-1617:21:164.6629002021-06-1617:21:174.7021502021-06-1617:21:184.706076Name:houston.ro.REMOTE_AI[22],dtype:float64,}The value under key"valve"in the output above is an instance ofpandas.Series, indented for readability.Advanced UsageCustom function calls on single Tag value>>>custom_operations={..."md5":lambdaa:hashlib.md5(str(a).encode()).hexdigest(),...}>>>tag_resolver=TagResolver(get_latest,custom_operations)>>>specs={..."pressure":Tag(METER_A),..."pressure_md5":Tag(METER_B).calc("md5"),...}>>>tag_resolver.series(specs){'pressure':4.15,'pressure_md5':'7fd3...'}Custom function calls on multiple Tag values>>>custom_operations={"max":max,"sum":lambda*vals:sum(vals)}>>>tag_resolver=TagResolver(get_latest,custom_operations)>>>specs={..."pressure_1":Tag(METER_A),..."pressure_2":Tag(METER_B),..."pressure_3":Tag(METER_C),..."pressure_highest":Tag.apply("max",Tag(METER_A),Tag(METER_B),Tag(METER_C)),..."highest_to_total_ratio":(...Tag.apply("max",Tag(METER_A),Tag(METER_B),Tag(METER_C))../Tag.apply("sum",Tag(METER_A),Tag(METER_B),Tag(METER_C))...),...}>>>tag_resolver.series(specs){'pressure_1':30.95,'pressure_2':19.1,'pressure_3':20.05,'pressure_highest':30.95,'highest_to_total_ratio':0.4415121255349501}Operations on SeriesThis example below is intentionally as similar as possible to the previous example. The only difference is the retrieval
function passed toTagResolver:get_seriesin this example vsget_latestin the previous one.See definition ofget_seriesandget_latestat the start ofFull Examplessection above.>>>custom_operations={"max":max,"sum":lambda*vals:sum(vals)}>>>tag_resolver=TagResolver(get_series,custom_operations)>>>specs={..."pressure_1":Tag(METER_A),..."pressure_2":Tag(METER_B),..."pressure_3":Tag(METER_C),..."pressure_highest":Tag.apply("max",Tag(METER_A),Tag(METER_B),Tag(METER_C)),..."highest_to_total_ratio":(...Tag.apply("max",Tag(METER_A),Tag(METER_B),Tag(METER_C))../Tag.apply("sum",Tag(METER_A),Tag(METER_B),Tag(METER_C))...),...}>>>tag_resolver.series(specs){'pressure_1':2021-06-1617:23:3748.302021-06-1617:23:3848.252021-06-1617:23:3948.15...Name:houston.ro.REMOTE_AI[3],dtype:float64,'pressure_2':2021-06-1617:23:3742.302021-06-1617:23:3842.102021-06-1617:23:3942.10...Name:houston.ro.REMOTE_AI[4],dtype:float64,'pressure_3':2021-06-1617:23:37119.952021-06-1617:23:38120.202021-06-1617:23:39120.10...Name:houston.ro.REMOTE_AI[5],dtype:float64,'pressure_highest':2021-06-1617:23:37119.952021-06-1617:23:38120.202021-06-1617:23:39120.10...dtype:float64,'highest_to_total_ratio':2021-06-1617:23:370.5696982021-06-1617:23:380.5708862021-06-1617:23:390.570953...dtype:float64,}The value in the output dicts above are instances ofpandas.Series, indented and trimmed for readability.But, where are the DataFrames?!Results fromTagResolver.resolvecan be passed directly topd.DataFrameto get the expected dataframe.This is a natural fit for series results, e.g:>>># ...continuing from the previous example>>>data=tag_resolver.series(specs)>>>pd.DataFrame(data)pressure_1pressure_2pressure_3pressure_highesthighest_to_total_ratio2021-06-1617:42:5049.6539.55139.10139.100.6092862021-06-1617:42:5149.8539.80138.70138.700.6074012021-06-1617:42:5349.5039.45138.50138.500.608925...For single-value responses, the DataFrame will have a single row, and the call topd.DataFramewill require an index
in addition to the data dict:>>>data={# from a previous example...'pressure_1':30.95,...'pressure_2':19.1,...'pressure_3':20.05,...'pressure_highest':30.95,...'highest_to_total_ratio':0.4415121255349501,...}>>>pd.DataFrame(data,index=[0])# [0] or any one-element iterablepressure_1pressure_2pressure_3pressure_highesthighest_to_total_ratio030.9519.120.0530.950.441512Multiple Data Stores>>>specs={..."avg_value":Tag(METER_A,"average"),..."above_average":Tag(METER_A)>Tag(METER_A,"average"),...}>>>resolver=TagResolver(get_series,average=get_average)>>>resolver.series(specs){"avg_value":42,"above_average: <pd.Series of bool values, True for points that are above 42, False for others>,}Responses with mixed single-value and multi-value items can also be passed intopd.DataFrameconstructor. Pandas
will automatically repeat any single-value items across all rows in the new dataframe.If using multiple data store function, the results will likely have different indexes.pd.DataFrameconstructor will
create a new dataframe with a combined index. This can result in a sparsely filled dataframe (many cells havingnp.nanvalue).More complex exampleTo expand on the previous example, let us require the difference between average value and any actual value, but
only if the value is above average, otherwise set it to 0. For example, this is a dataframe that we want to get:value   avg_value   above_average  positive_diff
2021-06-16 17:42:10       49.65       43.21            True        6.43999
2021-06-16 17:42:20       44.85       43.21            True        1.64000
2021-06-16 17:42:30       41.50       43.21           False              0
2021-06-16 17:42:40       40.90       43.21           False              0
2021-06-16 17:42:50       41.50       43.21           False              0
2021-06-16 17:42:60       43.30       43.21            True       0.089999
...There are many ways to get this result, here are a few equivalent ones.specs={"value":Tag(METER_A),"avg_value":Tag(METER_A,"average"),"above_average":Tag(METER_A)>Tag(METER_A,"average"),"positive_diff":(Tag(METER_A)-Tag(METER_A,"average"))ifTag(METER_A)>Tag(METER_A,"average")else0,}defpositive_or_0(val):returnvalifval>0else0specs={"value":Tag(METER_A),"avg_value":Tag(METER_A,"average"),"above_average":Tag(METER_A)>Tag(METER_A,"average"),"positive_diff":(Tag(METER_A)-Tag(METER_A,"average")).calc(positive_or_0),}avg_spec=Tag(METER_A,"average")meter_spec=Tag(METER_A)above_spec=meter_spec>avg_specdiff_spec=meter_spec-avg_specpositive_diff_spec=diff_specifabove_specelse0specs={"value":meter_spec,"avg_value":avg_spec,"above_average":above_spec,"positive_diff":positive_diff_spec,}Provided Data StoresAll examples in this document use data stores (a.k.a. retrieval functions) that are provided incognite_synthetic_tags.data_storesmodule. These are prepared to work withTagResolver.Provided stores:series- returns multiple values for every tag, aspd.Seriesinstances.point- same asseriesexcept that it returns only the last point in each series.series_at_time- same asseriesexcept it takesat_timeandlookbehind_start_timeas an alternative
toendandstartarguments.point_at_time- same aspointexcept it takesat_timeandlookbehind_start_timeas an alternative
toendandstartarguments.Check docstrings for more details.TODOAdd better support for passing lambdas tocalcanapply, e.g.Tag(..).calc(some_value=lambda val: val + 42)InternallySynthetic Tagsis usingDataFrameandSeriesfrom Pandas. There is a small performance penalty
associated with this, so we should probably make the effort to work directly with response objects from Cognite SDK.Check the effects ofinclude_outside_pointson the provided data store functions.