Package: ragged
RaggedIntroductionRaggedis a library for manipulating ragged arrays as though they wereNumPyorCuPyarrays, following theArray API specification.For example, this is aragged/jagged array:>>>importragged>>>a=ragged.array([[[1.1,2.2,3.3],[]],[[4.4]],[],[[5.5,6.6,7.7,8.8],[9.9]]])>>>aragged.array([[[1.1,2.2,3.3],[]],[[4.4]],[],[[5.5,6.6,7.7,8.8],[9.9]]])The values are all floating-point numbers, soa.dtypeisfloat64,>>>a.dtypedtype('float64')buta.shapehas non-integer dimensions to account for the fact that some of
its list lengths are non-uniform:>>>a.shape(4,None,None)In general, aragged.arraycan have any mixture of regular and irregular
dimensions, thoughshape[0](the length) is always an integer. This convention
follows theArray API's specification forarray.shape,
which must be a tuple ofintorNone:array.shape:Tuple[Optional[int],...](Our use ofNoneto indicate a dimension without a single-valued size differs
from theArray API's intention of specifying dimensions ofunknownsize,
but it follows the technical specification.Array API-consuming libraries
can try usingRaggedto find out if they are ragged-ready.)All of the normal elementwise and reducing functions apply, as well as slices:>>>ragged.sqrt(a)ragged.array([[[1.05,1.48,1.82],[]],[[2.1]],[],[[2.35,2.57,2.77,2.97],[3.15]]])>>>ragged.sum(a,axis=0)ragged.array([[11,8.8,11,8.8],[9.9]])>>>ragged.sum(a,axis=-1)ragged.array([[6.6,0],[4.4],[],[28.6,9.9]])>>>a[-1,0,2]ragged.array(7.7)>>>a[a*10%2==0]ragged.array([[[2.2],[]],[[4.4]],[],[[6.6,8.8],[]]])All of the methods, attributes, and functions in theArray APIwill be
implemented forRagged, as well as conveniences that are not required by theArray API. Seeopen issues marked "todo"forArray APIfunctions that still need to be written (out of 120 in total).Raggedhas twodevicevalues,"cpu"(backed byNumPy) and"cuda"(backed byCuPy). Eventually, all operations will be identical for CPU and
GPU.ImplementationRaggedis implemented usingAwkward Array(code,docs), which is an array library for arbitrary
tree-like (JSON-like) data. Because of its generality,Awkward Arraycannot
follow theArray API—in fact, its array objects can't have separatedtypeandshapeattributes (the arraytypecan't be factorized).Raggedis
thereforeaspecializationofAwkward Arrayfor numeric data in fixed-length and
variable-length lists, andaformalizationto adhere to theArray APIand its fully typed
protocols.SeeWhy does this library exist?under theDiscussionstab for
more details.Raggedis a thin wrapper aroundAwkward Array, restricting it to ragged
arrays and transforming its function arguments and return values to fit the
specification.Awkward Array, in turn, is time- and memory-efficient, ready for big
datasets. Consider the following:importgc# control for garbage collectionimportpsutil# measure process memoryimporttime# measure timeimportmathimportraggedthis_process=psutil.Process()defmeasure_memory(task):gc.collect()start_memory=this_process.memory_full_info().ussout=task()gc.collect()stop_memory=this_process.memory_full_info().ussprint(f"memory:{(stop_memory-start_memory)*1e-9:.3f}GB")returnoutdefmeasure_time(task):gc.disable()start_time=time.perf_counter()out=task()stop_time=time.perf_counter()gc.enable()print(f"time:{stop_time-start_time:.3f}sec")returnoutdefmake_big_python_object():out=[]foriinrange(10000000):out.append([j*1.1forjinrange(i%10)])returnoutdefmake_ragged_array():returnragged.array(pyobj)defcompute_on_python_object():out=[]forrowinpyobj:out.append([math.sqrt(x)forxinrow])returnoutdefcompute_on_ragged_array():returnragged.sqrt(arr)Theragged.arrayis 3 times smaller:>>>pyobj=measure_memory(make_big_python_object)memory:2.687GB>>>arr=measure_memory(make_ragged_array)memory:0.877GBand a sample calculation on it (square root of each value) is 50 times faster:>>>result=measure_time(compute_on_python_object)time:4.180sec>>>result=measure_time(compute_on_ragged_array)time:0.082secAwkward ArrayandRaggedare generally smaller and faster than their
Python equivalents for the same reasons thatNumPyis smaller and faster
than Python lists. SeeAwkward Arraypapers and presentationsfor more.InstallationRaggedis on PyPI:pipinstallraggedand will someday be on conda-forge.raggedis a pure-Python library that only depends onawkward(which, in
turn, only depends onnumpyand a compiled extension). In principle (i.e.
eventually),raggedcan be loaded into Pyodide and JupyterLite.AcknowledgementsSupport for this work was provided by NSF grantOAC-2103945and the
gracious help ofAwkward Array contributors.