Package: drb-driver-odata
OData driverThis drb-driver-odata module implements the OData protocol access
following the Copernicus Space Component schema with DRB
data model. It is able to navigate among Product entities of a OData
service.NodesODataServiceNodeRepresents the OData service. There are 3 type of serviceODataServiceNodeCSC,ODataServiceNodeDhusandODataServiceNodeDias. These nodes has no attribute and
has as children Product entities of the service defined byProductNode.
A specificProductNodecan be retrieved during the bracket and
slash navigation by hisName(str) or by hisId(UUID).OData services required can be access using the second (optional)
parameterauthofODataServiceNode, this parameter must be anrequests.auth.AuthBaseobject, seerequests documentationfor more details.ODataProductNodeRepresents aProductentity of the OData service. This
node has as attribute properties of the associated entity and has
for unique child aProductAttributeNodeODataProductAttributeNodeThis node allowing to represent the navigation link between theProductentity and its attributes. It has no attribute and has as
childrenAttributeentities associated to theProductentity,
defined byAttributeNodeODataAttributeNodeRepresents anAttributeentity. This node has no child and has as
attribute properties associated to theAttributeentity.PredicateODataCustomQueryThis predicate allows to retrieve a specific subset of children of anODataServiceNode.Cache set upSome cache has been implemented to limit the number of requests to the server.
These cache evicts cache entries based on both time and space.
the time and the cache size can be changed by adding these two variables to the runtime environment.DRB_ODATA_NODE_REQUEST_CACHE_EXPIRE_TIME_SEC=120DRB_ODATA_NODE_REQUEST_CACHE_MAX_ELEMENTS=32Here are all the queries that use a cachedef req_svc(odata: OdataNode) -> dict:
def req_svc_products(odata: OdataNode, **kwargs) -> list:
def req_product_by_uuid(odata: OdataNode, prd_uuid: str) -> dict:
def req_product_attributes(odata: OdataNode, prd_uuid: str) -> List[dict]:The time of the cache eviction can be change by callingreset_cache_expiration_time(sec=1):req_svc_products.reset_expiration_time(sec=1)Installationpip install drb-driver-odataExamplesfromuuidimportUUIDfromdrb.drivers.odataimportODataQueryPredicate,ExpressionFunc,ExpressionTypefromdrb.topicsimportresolver# Add '+odata' for recognize the odata driver usage.url='https+odata://my.csc.odata.com'# generate ODataServiceNode corresponding to the service.odata=resolver.create(url)# total number of childrenproduct_count=len(odata)# retrieve first ODataProductNodenode_idx=odata[0]# retrieve last ODataProductNodenode_idx=odata[-1]# retrieve 10 first productsproducts=odata[:10]# retrieve Product by namename='S2B_OPER_MSI_L0__GR_EPAE_..._D05_N02.06.tar'node_name_list=odata[name]# returns a listnode_name=odata[name]# returns first occurrence of the list# retrieve Product by UUIDuuid=UUID('0723d9bf-02a2-3e99-b1b3-f6d81de84b62')node_uuid=odata[uuid]# get product attributesprd_node=odata[uuid]attr_node=prd_node['Attributes']['Footprint']attr_type=attr_node@'ValueType'attr_value=attr_node.value# filter and order productsfiltered_children=odata[ODataQueryPredicate(filter="startswith(Name,'S1')",order="ContentLength desc")]# You can also use the Expression given with this driverfiltered_children=odata[ODataQueryPredicate(filter=ExpressionFunc.startswith(ExpressionType.property('Name'),'S1'),order=(ExpressionType.property('ContentLength'),ExpressionType.property('desc')))]The same example with DHus catalog (https://scihub.copernicus.eu/dhus/odata/v2)The only change is# get product attributes# For DhuS ValueType not exist in Attributes...attr_type=attr_node@'ContentType'A similar example with ONDA-DIAS catalog (https://catalogue.onda-dias.eu/dias-catalogue)importuuidfromdrb.drivers.odataimportODataQueryPredicatefromdrb.topicsimportresolverurl='https+odata://catalogue.onda-dias.eu/dias-catalogue'# generate ODataServiceNode corresponding to the service.odata=resolver.create(url)# retrieve Product by UUIDuuid_node=uuid.UUID('34a0a4ed-0246-4a57-827d-70350b96d03d')node_uuid=odata[uuid_node]attr_node=node_uuid['Attributes']children=attr_node.childrenfoot_print=node_uuid.get_attribute('footprint')print(foot_print)print(attr_node['Online quality check'].value)# search by product type limited two the 2 first resultfiltered_children=odata[ODataQueryPredicate(search='"(platformName:Sentinel-2) AND (productType:S2MSI2A)"',top="2")]Odata Query ExpressionTo help the user can create query by usingExpressionType,ExpressionOperatorandExpressionFuncExpressionTypeFunctionDescriptionExamplequeryboolTo put a boolean value in a queryExpressionType.bool(True)truestringTo put a string value in a queryExpressionType.string('toto')'toto'numberTo put a int or float value in a queryExpressionType.number(100)101collectionTo put an array of values in a queryExpressionType.collection([1,2,3])[1,2,3,4,5]propertyTo put an odata property in a queryExpressionType.property('Name')NamefootprintTo put an odata footprint in a queryExpressionType.footprint([(-12, 34), (32, 34))]geography'SRID=4326;Polygon((-12 34,32 34)ComparisonOperatorFunctionDescriptionExampleQueryeqequalComparisonOperator.eq(ExpressionType.property('My_Prop'),ExpressionType.string('Toto'))My_Prop eq 'Toto'neNot equalComparisonOperator.ne(ExpressionType.property('My_Prop'),ExpressionType.number(100))My_Prop ne 100hasHas flagsComparisonOperator.has(ExpressionType.property('Style'),ExpressionType.property("Sales.Color'Yellow'"))Style has Sales.Color'Yellow'co_inIs a member ofComparisonOperator.co_in(ExpressionType.property('Address/City'),GroupingOperator.group(ExpressionType.property("'Redmond', 'London'")))Address/City in ('Redmond', 'London')ltLess thanComparisonOperator.lt(ExpressionType.property('Price'),ExpressionType.number(20)Price lt 20leless than or equalComparisonOperator.le(ExpressionType.property('Price'),ExpressionType.number(20)Price le 20gtgreater thanComparisonOperator.gt(ExpressionType.property('Price'),ExpressionType.number(20)Price gt 20gegreater or equalComparisonOperator.ge(ExpressionType.property('Price'),ExpressionType.number(20)Price ge 20LogicalOperatorFunctionDescriptionExampleQuerylo_andLogical andLogicalOperator.lo_and(ge, gt)Price ge 20 and Price gt 20lo_orLogical orLogicalOperator.lo_or(ge, gt)Price ge 20 or Price gt 20lo_notLogical negationLogicalOperator.lo_not(ge)not Price ge 20ArithmeticOperatorFunctionDescriptionExampleQueryaddAdditionArithmeticOperator.add(ExpressionType.property('Price'),ExpressionType.number(5))Price add 20subSubtractionArithmeticOperator.sub(ExpressionType.property('Price'),ExpressionType.number(5))Price sub 20mulMultiplicationArithmeticOperator.mul(ExpressionType.property('Price'),ExpressionType.number(5))Price mul 20divDivisionArithmeticOperator.div(ExpressionType.property('Price'),ExpressionType.number(5))Price div 20divbyDecimal DivisionArithmeticOperator.divby(ExpressionType.property('Price'),ExpressionType.number(5))Price divby 20modModuloArithmeticOperator.mod(ExpressionType.property('Price'),ExpressionType.number(5))Price mod 20GroupingOperatorFunctionDescriptionExampleQuerygroupPrecedence groupingGroupingOperator.group(ExpressionOperator.add(ExpressionType.property('Price'),ExpressionType.number(5)))(Price add 5)ExpressionFuncfunctionExampleQueryconcatExpressionFunc.concat(ExpressionType.property('City'),ExpressionType.string(', '))concat(City,', ')containsExpressionFunc.contains(ExpressionType.property('City'),'London')contains(City,'London')endswithExpressionFunc.endswith(ExpressionType.property('City'),'on')endswith(City,'on')startswithExpressionFunc.startswith(ExpressionType.property('City'),'Lo')startswith(City,'Lo')indexofExpressionFunc.indexof(ExpressionType.property('Company'),'Gael')indexof(Company,'Gael')lengthExpressionFunc.length(ExpressionType.property('Company'))length(Company)substringExpressionFunc.substring(ExpressionType.property('Company'),1)substring(Company,1)hassubsetExpressionFunc.hassubset(ExpressionType.array([1, 2, 3, 4, 5]), ExpressionType.array([3, 4]))hassubset([1,2,3,4,5],[3,4])hassubsequenceExpressionFunc.hassubsequence(ExpressionType.array([1, 2, 3, 4, 5]), ExpressionType.array([3, 5]) )hassubsequence([1,2,3,4,5],[3,5])matchesPatternExpressionFunc.matchesPattern(ExpressionType.property('CompanyName'),ExpressionType.string('%5EA.*e$'))matchesPattern(CompanyName,'%5EA.*e$')tolowerExpressionFunc.tolower(ExpressionType.property('CompanyName'))tolower(CompanyName)toupperExpressionFunc.toupper(ExpressionType.property('CompanyName'))toupper(CompanyName)trimExpressionFunc.trim(ExpressionType.property('CompanyName'))trim(CompanyName)dayExpressionFunc.day(ExpressionType.property('StartTime'))day(StartTime)dateExpressionFunc.date(ExpressionType.property('StartTime'))date(StartTime)secondExpressionFunc.second(ExpressionType.property('StartTime'))second(StartTime)hourExpressionFunc.hour(ExpressionType.property('StartTime'))hour(StartTime)minuteExpressionFunc.minute(ExpressionType.property('StartTime'))minute(StartTime)monthExpressionFunc.month(ExpressionType.property('StartTime'))month(StartTime)timeExpressionFunc.time(ExpressionType.property('StartTime'))time(StartTime)totaloffsetminutesExpressionFunc.totaloffsetminutes(ExpressionType.property('StartTime'))totaloffsetminutes(StartTime)totalsecondsExpressionFunc.totalseconds(ExpressionType.property('StartTime'))totalseconds(StartTime)yearExpressionFunc.year(ExpressionType.property('StartTime'))year(StartTime)maxdatetimeExpressionFunc.maxdatetime(ExpressionType.property('StartTime'))maxdatetime(StartTime)mindatetimeExpressionFunc.mindatetime(ExpressionType.property('StartTime'))mindatetime(StartTime)nowExpressionFunc.now()now()ceilingExpressionFunc.ceiling(ExpressionType.property('Freight'))ceiling(Freight)floorExpressionFunc.floor(ExpressionType.property('Freight'))floor(Freight)roundExpressionFunc.round(ExpressionType.property('Freight'))round(Freight)castExpressionFunc.cast(ExpressionType.property('ShipCountry'),ExpressionType.property('Edm.String'))cast(ShipCountry,Edm.String)isofExpressionFunc.isof(ExpressionType.property('ShipCountry'),ExpressionType.property('Edm.String'))isof(ShipCountry,Edm.String)geo_distanceExpressionFunc.geo_distance(ExpressionType.property('CurrentPosition'),ExpressionType.property('TargetArea'))geo.distance(CurrentPosition,TargetArea)geo_intersectsExpressionFunc.geo_intersects(ExpressionType.property('CurrentPosition'),ExpressionType.property('TargetArea'))geo.intersects(CurrentPosition,TargetArea)csc_intersectExpressionFunc.csc_intersect(ExpressionType.property('CurrentPosition'),ExpressionType.property('TargetArea'))OData.CSC.Intersects(CurrentPosition,TargetArea)geo_lengthExpressionFunc.geo_length(ExpressionType.property('DirectRoute'))geo.length(DirectRoute)caseExpressionFunc.case([(ExpressionOperator.gt(ExpressionType.property('X'),ExpressionType.number(0)),1),(ExpressionOperator.lt(ExpressionType.property('X'),ExpressionType.number(0)),-1),(ExpressionType.bool(True),0)])case(X gt 0:1,X lt 0:-1,true:0)anyExpressionFunc.any('a',ExpressionOperator.gt(ExpressionType.property('a/TotalPrice'),ExpressionType.number(100)))any(a:a/TotalPrice gt 100)allExpressionFunc.all('d' ExpressionOperator.gt(ExpressionType.property('d/TotalPrice'),ExpressionType.number(100)))all(d:d/TotalPrice gt 100)Geographic query argumentFor geographic query this driver supports:String query like:"geography'SRID=4326;Polygon((0.0 0.0,1.0 0.0,1.0 1.0,0.0 1.0,0.0 0.0))'"List of tuple representing coordinate:[(0,0),(1,0),(1,1),(0,1),(0,0)])A polygon:Polygon([[0,0],[1,0],[1,1],[0,1],[0,0]])Odata Query BuilderAnother way to build Odata queries consists in using thequery_builderhelper.Once you have provided the url of your catalog (CSS, DAS, Dhus, Dias), the Odata service node, provides a classquery_builderto build the query filter compatible with your catalogThis class enables to build complex Odata queries using methods
to filter products bydate (sensing date, publication date, ...)geographic area ( polygon)string ( product name)numerical parameters (cloud coverage, online)collection of product (Sentinel-1, 2 ...)Examplesfromdrb.drivers.odataimportODataQueryPredicate,OdataFactoryfromdatetimeimportdatetime,dateimportjson# Add '+odata' for recognize the odata driver usage.url='https+odata://my.csc.odata.com'# generate ODataServiceNode corresponding to the service.svc=OdataFactory().create(url)# retrieve the query filter builderqbuilder=svc.get_filterquery_builder()# to search for products acquired between two dates# you can use eitherqbuilder.add_date_filter(qbuilder.tag_product_sensing_date,("2023-11-01T00:00:00.0Z","2023-11-30T00:00:00.0Z"))# orqbuilder.add_product_sensing_date_filter(("2023-11-01T00:00:00.0Z","2023-11-30T00:00:00.0Z"))# orqbuilder.add_product_sensing_date_filter((datetime(2023,11,1),datetime(2023,11,30)))# to search for products whose product name start either by S1# or S2 and contains either OCN or MSIL2Aqbuilder.add_string_filter(tag=qbuilder.tag_product_name,startswith=("S1","S2"),contains=("OCN","MSIL2A"))# orqbuilder.add_product_name_filter(startswith=("S1","S2"),contains=("SLC","MSIL2A"))# to search for product intersecting a geographic polygon arealong_min=-8long_max=8lat_max=55lat_min=40qbuilder.add_geometry_filter(((long_min,lat_min),(long_max,lat_min),(long_max,lat_max),(long_min,lat_max),(long_min,lat_min)))# or using a geoJson coordinates syntaxAGeoJson=[[long_min,lat_min],[long_max,lat_min],[long_max,lat_max],[long_min,lat_max],[long_min,lat_min]]qbuilder.add_geometry_filter(AGeoJson)# to search for products by attributs (cloud coverage <=50% )qbuilder.add_attribute_parameters(qbuilder.tag_cloud_attribute,50,"=<")# orqbuilder.add_product_cloud_parameters(50,"=<")# to print the Odata filterprint(qbuilder.build_filter().filter)# exemple of result for CSC catalog"(startswith(Name,'S1') or startswith(Name,'S2')) and (contains(Name,'SLC') or contains(Name,'MSIL2A')) and ContentDate/Start gt 2023-08-01T00:00:00.0Z and ContentDate/Start lt 2023-09-01T00:00:00.0Z and OData.CSC.Intersects(area=geography'SRID=4326;POLYGON((-8.0 40.0,8.0 40.0,8.0 55.0,-8.0 55.0,-8.0 40.0))') and Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value eq 50.0)"# and for DIAS'(name:S1* OR name:S2*) AND (name:*SLC* OR name:*MSIL2A*) AND beginPosition:[2023-08-01T00:00:00.0Z TO *] AND beginPosition:[* TO 2023-09-01T00:00:00.0Z] AND footprint:"Intersects(POLYGON((-8.0 40.0,8.0 40.0,8.0 55.0,-8.0 55.0,-8.0 40.0)))" AND cloudCoverPercentage:[* TO 50.0]'# and finally to apply your filters and get the list of wanted products, put the predicate in your odata service noderesult=svc[qbuilder.build_filter()]fornodeinresult:print(node.name)Another useful methods of the classquery_builderare:# to reset your query builder contentqbuilder.clear_filter()# to set query filters using a unique methodqbuilder.query_builder(sensingdate=(datetime(2023,8,1),datetime(2023,9,1)),footprint=AGeoJson,mission=ProductCollection.Sentinel3)