Package: icoscp_core
icoscp_coreA foundational ICOS Carbon Portal core products Python library for metadata and data access, designed to work with multiple data repositories who use ICOS Carbon Portal core server software stack to host and serve their data. At the moment, three repositories are supported:ICOS,SITES, andICOS Cities.Design goalsgood alignment with the server APIsoffer basic functionality, but in a robust way, and without sacrifices in performanceavoid unnecessary dependencies (only depend onnumpyand a small librarydacite), but aim for good integration withpandasprovide a solid foundation for future versions oficoscp—an ICOS-specific meta- and data access library developed by the Elaborated Products teamextensive use of type annotations and Python data classes, to safeguard agains preventable bugs, both in the library itself, and in the tools and apps written on top of it; a goal is to satisfy the typechecker in strict modeusage of autogenerated data classes produced from Scala back end code representing various metadata entities (e.g. data objects, stations) and their partssimultaneous support of three cross-cutting concerns:multiple repositories (ICOS, SITES, ICOS Cities)multiple ways of authenticationdata access through the HTTP API (on an arbitrary machine) and through file system (on a Jupyter notebook with "backdoor" data access); in the latter case the library is responsible for reporting the data usage event.Getting startedThe library is available on PyPI, can be installed withpip:$pipinstallicoscp_coreThe code examples below are usually provided for ICOS. For other Repositories (SITES or ICOS Cities), in the import directives, useicoscp_core.sitesoricoscp_core.cities, respectively, instead oficoscp_core.icos.AuthenticationMetadata access does not require authentication, and is achieved by a simple import:fromicoscp_core.icosimportmetaWhen using the library on an accordingly configured Jupyter notebook service hosted by the ICOS Carbon Portal, authentication is not required when using two of the data access methods:get_columns_as_arraysbatch_get_columns_as_arraysavailable ondataimport fromicoscp_core.icospackage. When using other data access methods, or when running the code outside ICOS Jupyter environment, or if the Jupyter environment has not been provisioned with filesystem data access to your Repository, all data access methods require authentication.Authentication can be initialized in a number of ways.Credentials and token cache file (default)This approach should only be used on machines the developer trusts.A username/password account with the respective authentication service (links for:ICOS,SITES,ICOS Cities) is required for this. Obfuscated (not readable by humans) password is stored in a file on the local machine in a default user-specific folder. To initialize this file, run the following code interactively (only needs to be once for every machine):fromicoscp_core.icosimportauthauth.init_config_file()After the initialization step is done, access to the metadata and data services is achieved by a simple import:fromicoscp_core.icosimportmeta,dataAs an alternative, the developer may choose to use a specific file to store the credentials and token cache. In this scenario,dataservice needs to be initialized as follows:fromicoscp_core.icosimportbootstrapauth,meta,data=bootstrap.fromPasswordFile("<desired path to the file>")# the next line needs to be run interactively (only once per file)auth.init_config_file()Static authentication token (prototyping)This option is good for testing, on a public machine or in general. Its only disadvantage is that the tokens have limited period of validity (100000 seconds, less than 28 hours), but this is precisely what makes it acceptable to include them directly in the Python source code.The token can be obtained from the "My Account" page (links for:ICOS,SITES,ICOS Cities), which can be accessed by logging in using one of the supported authentication mechanisms (username/password, university sign-in, OAuth sign in). After this the bootstrapping can be done as follows:fromicoscp_core.icosimportbootstrapcookie_token='cpauthToken=WzE2OTY2NzQ5OD...'meta,data=bootstrap.fromCookieToken(cookie_token)Explicit credentials (advanced option)The user may choose to use their own mechanism of providing the credentials to initialize the authentication. This should be considered as an advanced option.(Please do not put your password as clear text in your Python code!)This can be achieved as follows:fromicoscp_core.icosimportbootstrapmeta,data=bootstrap.fromCredentials(username_variable,password_containing_variable)Metadata accessMetadata access requires no authentication, and is performed using an instance ofMetadataClientclass easily obtainable through an import:fromicoscp_core.icosimportmetaAn important background information is that all the metadata-represented entities (data objects, data types, documents, collections, measurement stations, people, etc) are identified by URIs. The metadata-access methods usually accept these URIs as input arguments, and the returned values tend to be instances ofPython dataclasses, which brings:better syntax in comparison with generic dictionaries (dot-notation attribute access instead of dictionary value access, for exampledobj_meta.specification.project.self.uriinstead ofdobj_meta["specification"]["project"]["self"]["uri"])autocomplete of the dataclass attributes (works even in Jupyter notebooks)type checking, when developing with type annotations and a type checker (typically available from an IDE, but not from Jupyter)The following code showcases the main metadata access methods.fromicoscp_core.icosimportmeta,ATMO_STATIONfromicoscp_core.metaclientimportTimeFilter,SizeFilter,SamplingHeightFilter# fetches the list of known data types, including metadata associated with themall_datatypes=meta.list_datatypes()# data types with structured data accesspreviewable_datatypes=[dtfordtinall_datatypesifdt.has_data_access]# fetch lists of stations, with basic metadataicos_stations=meta.list_stations()atmo_stations=meta.list_stations(ATMO_STATION)all_known_stations=meta.list_stations(False)# get detailed metadata for a stationhtm_uri='http://meta.icos-cp.eu/resources/stations/AS_HTM'htm_station_meta=meta.get_station_meta(htm_uri)# list data objects with basic metadata# a contrived, complicated example to demonstrate the possibilities# all the arguments are optional# see the Python help for the method for more detailsfiltered_atc_co2=meta.list_data_objects(datatype=["http://meta.icos-cp.eu/resources/cpmeta/atcCo2L2DataObject","http://meta.icos-cp.eu/resources/cpmeta/atcCo2NrtGrowingDataObject"],station="http://meta.icos-cp.eu/resources/stations/AS_GAT",filters=[TimeFilter("submTime",">","2023-07-01T12:00:00Z"),TimeFilter("submTime","<","2023-07-10T12:00:00Z"),SizeFilter(">",50000),SamplingHeightFilter("=",216)],include_deprecated=True,order_by="fileName",limit=50)# get detailed metadata for a single data objectdobj_uri='https://meta.icos-cp.eu/objects/BbEO5i3rDLhS_vR-eNNLjp3Q'dobj_meta=meta.get_dobj_meta(dobj_uri)Detailed help on the available metadata access methods can be obtained fromhelp(meta)call.Repository-specific functionalityThe majority of functionality of the library is common to all the supported data Repositories. However, in some cases Repository-specific reusable code may be useful. Such code is planned to be placed into corresponding packages. There is only one example of such code at the moment:fromicoscp_core.icosimportstation_class_lookuphtm_uri='http://meta.icos-cp.eu/resources/stations/AS_HTM'htm_class=station_class_lookup()[htm_uri]Data accessAfter having identified an interesting data object or a list of objects in the previous step, one can access their data content in a few ways. Data access is provided by an instance ofDataClientclass most easily obtained by importfromicoscp_core.icosimportdataThe following are code examples showcasing the main data access methods.Downloading original data object contentGiven basic data object metadata (or just the URI id) one can download the original data to a folder like so:filename=data.save_to_folder(dobj_uri,'/myhome/icosdata/')The method requires authentication, even on ICOS Jupyter instances. Works on all data objects (all kinds, and regardless of variable metadata availability)Station-specific time seriesStation-specific time series, that have variable metadata associated with them, enjoy a higher level of support. The variables with metadata representation (which may be only a subset of the variables present in the original data) can be efficiently accessed using this library. For single-object access, a complete data object metadata is required. The output can be readily converted to a pandasDataFrame, but can be used as is (a dictionary of numpy arrays).  It is possible to explicitly limit variables for access, and to slice the time series.Authentication may be optional on ICOS Jupyter instances.importpandasaspd# get dataset columns as typed arrays, ready to be imported into pandasdobj_arrays=data.get_columns_as_arrays(dobj_meta,['TIMESTAMP','co2'])df=pd.DataFrame(dobj_arrays)One way to distinguish the objects with structured data access is that their data types (used for filtering the data objects, see the metadata access section) havehas_data_accessproperty equal toTrue.Batch data accessIn many scripting scenarios, data objects are processed in batches of uniform data types. In this case, rather than usingget_columns_as_arraysmethod in a loop, it is much more efficient to use a special batch-access method. This will significantly reduce the number of round trips to the HTTP metadata service, greatly speeding up the operation:multi_dobjs=data.batch_get_columns_as_arrays(filtered_atc_co2,['TIMESTAMP','co2'])wherefiltered_atc_co2is a either a list from the metadata examples above, or just a list of plain data object URI IDs. The returned value is a generator of pairs, where first value is the basic data object metadata (or just a plain URI id, depending on what was used as the argument), and the second value is the same as the return value fromget_columns_as_arraysmethod (a dictionary of numpy arrays, with variable names as keys)If it is desirable to convert the data to pandasDataFrames, it can be done like so:importpandasaspdmulti_df=((dobj,pd.DataFrame(arrs))fordobj,arrsinmulti_dobjs)CSV representation accessThe data server offers (partial) CSV representations for fully-supported time series datasets. That service can be used from this library as follows:importpandasaspdcsv_stream=data.get_csv_byte_stream(dobj_uri)df=pd.read_csv(csv_stream)but usingget_columns_as_arraysandbatch_get_columns_as_arraysis to be preferred for performance reasons, especially on ICOS Jupyter instances. Authentication is always required to use this method.Advanced metadata access (SPARQL)For general metadata enquiries not offered by the API explicitly, it is often possible to design a SPARQL query that would provide the required information. The query can be run withsparql_selectmethod ofMetadataClient, and the output of the latter can be parsed using "as_<rdf_datatype>"-named methods inicoscp_core.sparqlmodule. For example:fromicoscp_core.icosimportmetafromicoscp_core.sparqlimportas_string,as_uriquery="""prefix cpmeta: <http://meta.icos-cp.eu/ontologies/cpmeta/>select *from <http://meta.icos-cp.eu/documents/>where{?doc a cpmeta:DocumentObject .FILTER NOT EXISTS {[] cpmeta:isNextVersionOf ?doc}?doc cpmeta:hasDoi ?doi .?doc cpmeta:hasName ?filename .}"""latest_docs_with_dois=[{"uri":as_uri("doc",row),"filename":as_string("filename",row),"doi":as_string("doi",row)}forrowinmeta.sparql_select(query).bindings]