Package: vineyard-io
vineyard-io: IO drivers forvineyardvineyard-io is a collection of IO drivers forvineyard. Currently it supportsLocal filesystemAWS S3Aliyun OSSHadoop filesystemThe vineyard-io package leverages thefilesystem-specto support other storage sinks and sources in a unified fashion. Other adaptors that works for fsspec
could be plugged in as well.IO AdaptorsVineyard has a set of prebuilt IO adaptors, that can serve as common routines for
various IO operations and can take place of boilerplate parts in computation tasks.Vineyard is capable of reading from and writing data to multiple file systems.
Behind the scene, it leveragefsspecto delegate the workload to various file system implementations.Specifically, we can specify parameters to be passed to the file system, through thestorage_optionsparameter.storage_optionsis a dict that pass additional keywords to the file system,
For instance, we could combinepath=hdfs:///path/to/filewithstorage_options = {"host": "localhost", "port": 9600}to read from a HDFS.Note that you must encode thestorage_optionsby base64 before passing it to the scripts.Alternatively, we can encode such information into the path,
such as:hdfs://<ip>:<port>/path/to/file.To read from multiple files you can pass a glob string or a list of paths,
with the caveat that they must all have the same protocol.Their functionality are described as follows:read_bytesUsage: vineyard_read_bytes <ipc_socket> <path> <storage_options> <read_options> <proc_num> <proc_index>Read a file on local file systems, OSS, HDFS, S3, etc. toByteStream.write_bytesUsage: vineyard_write_bytes <ipc_socket> <path> <stream_id> <storage_options> <write_options> <proc_num> <proc_index>Write aByteStreamto a file on local file system, OSS, HDFS, S3, etc.read_orcUsage: vineyard_read_orc <ipc_socket> <path/directory> <storage_options> <read_options> <proc_num> <proc_index>Read a ORC file on local file systems, OSS, HDFS, S3, etc. toDataframeStream.write_orcUsage: vineyard_read_orc <ipc_socket> <path/directory> <storage_options> <read_options> <proc_num> <proc_index>Write aDataframeStreamto a ORC file on local file system, OSS, HDFS, S3, etc.read_vineyard_dataframeUsage: vineyard_read_vineyard_dataframe <ipc_socket> <vineyard_address> <storage_options> <read_options> <proc num> <proc index>Read aDataFramein vineyard as aDataframeStream.write_vineyard_dataframeUsage: vineyard_write_vineyard_dataframe <ipc_socket> <stream_id> <proc_num> <proc_index>Write aDataframeStreamto aDataFramein vineyard.serializerUsage: vineyard_serializer <ipc_socket> <object_id>Serialize a vineyard object (non-global or global) as aByteStreamor a set ofByteStream(StreamCollection).deserializerUsage: vineyard_deserializer <ipc_socket> <object_id>Deserialize aByteStreamor a set ofByteStream(StreamCollection) as a vineyard object.read_bytes_collectionUsage: vineyard_read_bytes_collection <ipc_socket> <prefix> <storage_options> <proc_num> <proc_index>Read a directory (on local filesystem, OSS, HDFS, S3, etc.) as aByteStreamor a set ofByteStream(StreamCollection).write_bytes_collectionUsage: vineyard_write_vineyard_dataframe <ipc_socket> <stream_id> <proc_num> <proc_index>Write aByteStreamor a set ofByteStream(StreamCollection) to a directory (on local filesystem, OSS, HDFS, S3, etc.).parse_bytes_to_dataframeUsage: vineyard_parse_bytes_to_dataframe.py <ipc_socket> <stream_id> <proc_num> <proc_index>Parse aByteStream(in CSV format) as aDataframeStream.parse_dataframe_to_bytesUsage: vineyard_parse_dataframe_to_bytes <ipc_socket> <stream_id> <proc_num> <proc_index>Serialize aDataframeStreamto aByteStream(in CSV format).dump_dataframeUsage: vineyard_dump_dataframe <ipc_socket> <stream_id>Dump the content of aDataframeStream, for debugging usage.