Package: aos
And-Or Shape (aos) LanguageWriting data pipelines involves complex data transformations over nested data, e.g., list of dictionaries or dictionary of tensors.Theshapeof nested data is not explicit in code and hence not accessible readily to the developer.Leads to cognitive burden (guessing shapes), technical debt and inadvertent programming errors.Data pipelines are very opaque to examination and comprehension.aosis a compact, regex-like language for describing the shapes (schemas) of both homogeneous (tensors) and heterogeneous (dictionaries, tables) data, and combinations, independent of the specific data library.Based on an intuitiveregex-likealgebra of data shapes.Inferaosshape from a data instance:aos.infer.infer_aos.Validatedata againstaosshapes anywhere:aos.checker.instanceof.Transformdata usingaosshapes, declaratively:aos.tfm.do_tfm.Allows writing explicit data shapes,inlinein code. In Python, use type annotations.Write shapes for a variety of data conveniently -- Python native objects (dict,list, scalars), tensors (numpy,pytorch,tf),pandas,hdf5,tiledb,xarray,struct-tensor, etc.Installationpip install aosShape of Data ?Consider a few quick examples.the shape of scalar data is simply its type, e.g.,int,float,str, ...for nested data, eg.  list ofints:(int)*for a dictionary of form{'a': 3, b: 'hi'}: shape is(a & int) | (b & str).Now, we can describe the shape ofarbitrary, nesteddata with these&(and)-|(or) expressions. Intuitively, a list is anor-structure, a dictionary is anorofands, a tensor is anand-structure, and so on.Why is alistan or-structure? Ask: how do weaccessany valuevin thelist? Choosesomeindex of the list, corresponding to the valuev.Similarly, adictionaryis an or-and structure: we pickoneof thekeys, together (and) with itsvalue.In contrast, an n-dimensionaltensorhas anand-shape: we must choose indices fromallthe dimensions of the tensor to access a scalar value.In general, for a data structure, weask: what choices must we make to access a scalar value?Thinking in terms ofand-orshapes takes a bit of practice initially. Read more about the and-or expressionshere.More complexaosexamplesLists over shapesare denoted as(s)*.  Shorthand for(s|..|s).Dictionary:(k1 & v1) | (k2 & v2) | ... | (kn & vn)wherekiandviis theith key and value.Pandas tables:(n & ( (c1&int)| (c2&str) | ... | (cn&str) )wherenis the row dimension (the number of rows) andc1,...,cnare column names.Theaosexpressions are verycompact. For example, consider a highly nested Python objectXof typeSequence[Tuple[Tuple[str, int], Dict[str, str]]]This is both verbose and hard to interpret. Instead,X'saosis written compactly as((str|int) | (str : str))*.The full data shape may be irrelevant in many cases. To keep it brief, the language supports wildcards:_and...to allow writing partial shapes.So, we could write a dictionary's shape as(k1 & ...)| ... | (kn & ...).Shape InferenceUnearthing the shape of opaque data instances, e.g., returned from a web request, or passed into a function call, is a major pain.Useaos.infer.infer_aosto obtain compact shapes of arbitrary data instances.From command line, runaos-infer <filename.json>fromaos.inferimportinfer_aosdeftest_infer():d={"checked":False,"dimensions":{"width":5,"height":10},"id":1,"name":"A green door","price":12.5,"tags":["home","green"]}infer_aos(d)# ((checked & bool)# | (dimensions & ((width & int) | (height & int)))# | (id & int) | (name & str) | (price & float) | (tags & (str *)))dlist=[]foriinrange(100):d['id']=idlist.append(d.copy())infer_aos(dlist)# ((checked & bool)# | (dimensions & ((width & int) | (height & int)))# | (id & int) | (name & str) | (price & float) | (tags & (str *)))*Shape/Schema ValidationUsingaos.checker.instanceof, we canwriteaosassertions to validate data shapes (schemas).validate data structure partially using placeholders:_matches a scalar,...matches an arbitrary object (sub-tree).works with python objects, pandas, numpy, ..., extensible to other data types (libraries).fromaos.checkerimportinstanceofdeftest_pyobj():d={'city':'New York','country':'USA'}t1=('Google',2001)t2=(t1,d)instanceof(t2,'(str | int) | (str & str)')#validinstanceof(t2,'... | (str & _)')#validinstanceof(t2,'(_ | _) | (str & int)')#errortlist=[('a',1),('b',2)]instanceof(tlist,'(str | int)*')#validdeftest_pandas():d={'id':'CS2_056','cost':2,'name':'Tap'}df=pd.DataFrame([d.items()],columns=list(d.keys()))instanceof(df,'1 & (id | cost | name)')deftest_numpy():#arr = np.array()arr=np.array([[1,2,3],[4,5,6]])instanceof(arr,'2 & 3')deftest_pytorch():#arr = np.array()arr=torch.tensor([[1,2,3],[4,5,6]])instanceof(arr,'2 & 3')Transformations with AOSBecauseaosexpressions can bothmatchandspecifyheterogeneous data shapes, we can writeaosrulestotransformdata.The rules are written aslhs -> rhs, where bothlhsandrhsareaosexpressions:lhsmatchesa part (sub-tree) of the input data instanceI.queryvariables in thelhscapture(bind with) parts ofI.rhsspecifies the expected shape (aos) of the output data instanceO.To write rules, ask: whichpartsofI, do we need to constructO?fromaos.tfmimportdo_tfmdeftfm_example():# input dataI={'items':[{'k':1},{'k':2},{'k':3}],'names':['A','B','C']}# specify transformation (left aos -> right aos)# using `query` variables `k` and `v`# here `k` binds with each of the keys in the list and# `v` binds with the corresponding value# the `lhs` automatically ignores parts of I, which are irrelevant to Otfm='items & (k & v)* -> values & (v)*'O=do_tfm(I,tfm)print(O)# {'values': [1, 2, 3]}The above example illustrates a simple JSON transformation usingaosrules. Rules can be more complex, e.g., includeconditions,functionapplication on query variables. They work not only with JSON data, but also apply to heterogeneous nested objects.See more exampleshereandhere.And-Or Shape DimensionsThe above examples of use strings or type names (str) or integer values (2,3) in shape expressions. A more principled approach is to first declaredimension namesand define shape over these names.Data is defined over two kinds of dimensions:Continuous. A range of values, e.g., a numpy array of shape (5, 200) is defined over two continuous dimensions, saynandd, wherenranges over values0-4anddranges over0-199.Categorical. A set of names, e.g., a dictionary{'a': 4, 'b': 5}is defined overkeys(dim names)['a', 'b']. One can also view each key, e.g.,aorb, as aSingletondimension.Programmatic API. The library provides an API to declare both type of dimensions andaosexpressions over these dimensions, e.g., declarenanddas two continuous dimensions and then define shapen & d.StatusThe library is under active development. More documentation coming soon..