Package: necstdb
NECSTDBDatabase for NECST.TL;DRNECST, an abbreviation ofNEw Control System for Telescope, is a flexible control system for radio telescopes. Its efficient data storage is provided here.The database contains tables, which keep individual topic of time series data with some metadata attached to them, e.g. spectral data from one spectrometer board (array of data + timestamp), weather data (temperature + humidity + wind speed + ... + timestamp), etc.FeaturesThis package provides:database writer with quite flexible data format supportdatabase reader which supports various output formatInstallationpipinstallnecstdbUsageDatabase WriterCreate new database instance>>>db_dir="path/to/new_database_directory">>>db=necstdb.opendb(db_dir,mode="w")# "w" stands for "write"Create table with data information>>>data1_info={..."data":[...{"key":"recorded_time","format":"d","size":8},...{"key":"n_scan_lines","format":"i","size":4},...{"key":"obsmode","format":"3s","size":3},...{"key":"timestamp","format":"d","size":8},...],..."memo":"generated by db_logger_operation",..."necstdb_version":necstdb.__version__,...}>>>db.create_table("data1",data1_info)The keys "key" and "format" in "data" list are required. The format characters and the sizes are listed below. For more information, see the struct moduledocumentationand/or the ROS messagewiki.Changed in v0.2.5: "size" in "data" list is no longer required, but is optional.The "memo" and "necstdb_version" keys are not necessary. You can also add any other keys to the data information dict.ROS formatFormat characterSize [byte]bool?1int8b1int16h2int32i4int64q8uint8B1uint16H2uint32I4uint64Q8float32f4float64d8string[length]s1 * lengthNOTE:Array data are also supported. To write them, use repeat count syntax (e.g.3d) or format character sequence syntax (e.g.ddd) for the "format", and the sum of the elements' sizes for the "size".{"key":"data_array","format":"3d","size":3*8}# repeat count syntax{"key":"data_array","format":"ddd","size":8+8+8}# format character sequence syntaxWrite data into table>>>table1=db.open_table("data1",mode="ab")# "ab" stands for "append binary">>>data1=[1.6294488758e9,3,b"SKY",1.6294488757e9]# string data are not allowed, use bytes instead>>>table1.append(*data1)Call theappendmethod every time you get new data.NOTE:Data to pass toappendmethod should be flattened. The nested structure will be reconstructed on reading.data=[1,2,[3,4,5],6,7]data=necstdb.utils.flatten_data(data)table.append(*data)NOTE:Multi-dimensional data are not supported.NOTE:Shape of array or length of string in every data must be the same as the one in data information. (Mismatch of string length won't raise an error, but stored data can be broken or incomplete.)Save files into the database>>>db.save_file("example.txt","Content of the file.")Database ReaderOpen the database instance>>>db=necstdb.opendb("path/to/database_directory")Read a desired topic>>>data1=db.open_table("data1").read(astype="array")>>>data1array([[1.6294488758e9,3,b'SKY',1.6294488757e9],...],dtype=[('recorded_time','<f8'),('n_scan_lines','<i4'),('obsmode','|S3'),('timestamp','<f8')])The supportedastypekeywords forreadmethod are:Output typeKeywordsNotestuple"tuple"*1dict"dict"pandas.DataFrame"pandas", "dataframe", "data_frame", "df"*2numpy.ndarray"array", "structuredarray", "structured_array", "sa"bytes"buffer", "raw"*1: Array data are not supported, but will be flattened.*2:Changed in v0.2.5: Keyworddfis now supported.Read files saved in the database>>>db.read_file("example.txt")"Content of the file.",""This method assumes the file can be read asstr. This attempt will fail for binary files, so useasbytesoption explicitly.>>>db.read_file("example_binary.data",asbytes=True)b"Content of the file.",b""This method returns 2strorbytesvalues. The second value is metadata for the file saved usingdb.save_file(filename, data, metadata)method.MiscList all the tables contained in the database>>>db=necstdb.opendb("path/to/database_directory")>>>db.list_tables()['data1','spectral_data','weather_data',...]Archive the database>>>db=necstdb.opendb("path/to/database_directory")>>>db.checkout(saveto="path/to/archive.tar.gz",compression="gz")Get informations of all tables in the database>>>db=necstdb.opendb("path/to/database_directory")>>>db.get_info()"""file size [byte]    #records    record size [byte]  formattable namedata1               2448                102         24                  di3sdspectral_data       41948160            320         131088              d32768fdweather_data        6328                113         56                  ddddddd"""Read particular columns and/or rows of the database>>>db=necstdb.opendb("path/to/database_directory")>>>data=db.open_table("data1").read(num=5,start=3,cols=["timestamp","obsmode"],astype="tuple")# order of cols won't be preserved((b'SKY',1.6294488788e9)# 4th element (caution 0-based indexing)(b'SKY',1.6294488798e9)(b'SKY',1.6294488808e9)(b'SKY',1.6294488818e9)(b'HOT',1.6294488828e9))Flatten nested array>>>data=[1,2,3,[4,5],6]>>>flattened=necstdb.utils.flatten_data(data)[1,2,3,4,5,6]