Package: arcae
RationaleThe structure of Apache Arrow Tables is highly similar to that of CASA TablesIt’s easy to convert Arrow Tables between many different languagesOnce in Apache Arrow format, it is easy to store data in modern, cloud-native disk formats such as parquet and orc.Converting CASA Tables to Arrow in the C++ layer avoids the GILAccess to non thread-safe CASA Tables is constrained to a ThreadPool containing a single threadIt also allows us to write astrometric routines in C++, potentially side-stepping thread-safety
and GIL issues with the CASA Measures server.Build Wheel LocallyIn the user or, even better, a virtual environment:$pipinstall-Upipcibuildwheel$bashscripts/run_cbuildwheel.sh-p3.10WarningOnly linux wheels are currently supported.Local DevelopmentIn the directory containing the source, setup your development environment as follows:$pipinstall-Upipvirtualenv$virtualenv-ppython3.10/venv/arcaedev$./venv/arcaedev/bin/activate(arcaedev)exportVCPKG_TARGET_TRIPLET=x64-linux-dynamic-cxx17-abi1-dbg(arcaedev)pipinstall-e.[test](arcaedev)exportLD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/vcpkg/installed/$VCPKG_TARGET_TRIPLET/lib(arcaedev)py.test-s-vvv--pyargsarcaeUsageExample Usage:importjsonfrompprintimportpprintimportarcaeimportpandasaspdimportpyarrowaspaimportpyarrow.parquetaspq# Obtain (partial) Apache Arrow Table from a CASA Tablecasa_table=arcae.table("/path/to/measurementset.ms")arrow_table=casa_table.to_arrow()# read entire tablearrow_table=casa_table.to_arrow(10,20)# startrow, nrowassertisinstance(arrow_table,pa.Table)# Print JSON-encoded Table and Column keywordspprint(json.loads(arrow_table.schema.metadata[b"__arcae_metadata__"]))pprint(json.loads(arrow_table.schema.field("DATA").metadata[b"__arcae_metadata__"]))# Extract Arrow Table columns into numpy arraystime=arrow_table.column("TIME").to_numpy()data=arrow_table.column("DATA").to_numpy()# currently, arrays of object arrays, overly slow and memory hungrydf=arrow_table.to_pandas()# currently slow, memory hungry due to arrays of object arrays# Write Arrow Table to parquet filepq.write_table(arrow_table,"measurementset.parquet")See the test cases for further use cases.Exporting Measurement Sets to Arrow Parquet DatasetsAn export script is available:$arcaeexport/path/to/the.ms--nrow50000$treeoutput.arrow/output.arrow/├──ANTENNA│└──data0.parquet├──DATA_DESCRIPTION│└──data0.parquet├──FEED│└──data0.parquet├──FIELD│└──data0.parquet├──MAIN│└──FIELD_ID=0│└──PROCESSOR_ID=0│├──DATA_DESC_ID=0││├──data0.parquet││├──data1.parquet││├──data2.parquet││└──data3.parquet│├──DATA_DESC_ID=1││├──data0.parquet││├──data1.parquet││├──data2.parquet││└──data3.parquet│├──DATA_DESC_ID=2││├──data0.parquet││├──data1.parquet││├──data2.parquet││└──data3.parquet│└──DATA_DESC_ID=3│├──data0.parquet│├──data1.parquet│├──data2.parquet│└──data3.parquet├──OBSERVATION│└──data0.parquetThis data can be loaded into an Arrow Dataset:>>>importpyarrowaspa>>>importpyarrow.datasetaspad>>>main_ds=pad.dataset("output.arrow/MAIN")>>>spw_ds=pad.dataset("output.arrow/SPECTRAL_WINDOW")LimitationsSome edge cases have not yet been implemented, but could be with some thought.Columns with unconstrained rank (ndim == -1) whose rows, in practice, have differing dimensions.
Unconstrained rank columns whose rows actually have the same rank are catered for.Not yet able to handle TpRecord columns. Probably simplest to convert these rows to json and store as a string.Not yet able to handle TpQuantity columns. Possible to represent as a run-time parametric Arrow DataType.to_numpy()conversion of nested lists produces nested numpy arrays, instead of tensors.
This ispossiblebut requires some changes to howC++ Extension Types are exposed in Python.EtymologyNoun:arcaf (genitivearcae); first declension
A chest, box, coffer, safe (safe place for storing items, or anything of a similar shape)Pronounced:ar-ki.