Patent ID: 10162836

Abstract:
Metadata associated with a plurality of sub-files associated with a single shared file is stored in a parallel file system. A plurality of processes generate a shared file. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file and metadata for the at least one portion of the shared file on one or more of the plurality of object storage servers. The compute node is further configured to store the metadata by striping the metadata across a plurality of subdirectories of the shared file. The metadata is optionally striped across the plurality of subdirectories in a round-robin manner. The plurality of subdirectories are stored on one or more of the object storage servers. Write and read processes optionally communicate using a message passing interface. A given write process optionally writes metadata for a given portion of the shared file to an index file in a particular one of the subdirectories corresponding to the given portion.