Package: gem5art-artifact
gem5art artifact packageThis package contains theArtifacttype and an artifact database for use withgem5art.Please cite thegem5art paperwhen using the gem5art packages.
This documentation can be found on thegem5 websitegem5art artifactsAll unique objects used during gem5 experiments are termed "artifacts" in gem5art.
Examples of artifacts include: gem5 binary, gem5 source code repo, Linux kernel source repo, linux binary, disk image, and packer binary (used to build the disk image).
The goal of this infrastructure is to keep a record of all the artifacts used in a particular experiment and to return the set of used artifacts when the same experiment needs to be performed in the future.The description of an artifact serves as the documentation of how that artifact was created.
One of the goals of gem5art is for these artifacts to be self contained.
With just the metadata stored with the artifact a third party should be able to perfectly reproduce the artifact.
(We are still working toward this goal.
For instance, we are looking into using docker to create artifacts to separate artifact creation from the host platform its run on.)Each artifact is characterized by a set of attributes, described below:command: command used to build this artifacttyp: type of the artifact e.g. binary, git repo etc.name: name of the artifactcwd: current working directory, where the command to build the artifact is runpath: actual path of the location of the artifactinputs: a list of the artifacts used to build the current artifactdocumentation: a docstring explaining the purpose of the artifact and any other useful information that can help to reproduce the artifactAdditionally, each artifact also has the following implicit information.hash: an MD5 hash for a binary artifact or a git hash for a git artifacttime: time of the creation of an artifactid: a UUID associated with the artifactgit: a dictionary containing the origin, current commit and the repo name for a git artifact (will be an empty dictionary for other types of artifacts)These attribute are not specified by the user, but are generated by gem5art automatically (when theArtifactobject is created for the first time).An example of how a user would create a gem5 binary artifact using gem5art is shown below.
In this example, the type, name, and documentation are up to the user of gem5art.
You're encouraged to use names that are easy to remember when you later query the database.
The documentation attribute should be used to completely describe the artifact that you are saving.gem5_binary=Artifact.registerArtifact(command='scons build/X86/gem5.opt',typ='gem5 binary',name='gem5',cwd='gem5/',path='gem5/build/X86/gem5.opt',inputs=[gem5_repo,],documentation='''Default gem5 binary compiled for the X86 ISA.This was built from the main gem5 repo (gem5.googlesource.com) withoutany modifications. We recently updated to the current gem5 masterwhich has a fix for memory channel address striping.''')Another goal of gem5art is to enable sharing of artifacts among multiple users, which is achieved through the use of the centralized database.
Basically, whenever a user tries to create a new artifact, the database is searched to find if the same artifact exists there.
If it does, the user can download the matching artifact for use.
Otherwise, the newly created artifact is uploaded to the database for later use.
The use of database also avoids running identical experiments (by generating an error message if a user tries to execute exact run which already exists in the database).Creating artifactsTo create anArtifact, you must useregisterArtifactas shown in the above example as well.
This is a factory method which will initially create the artifact.When callingregisterArtifact, the artifact will automatically be added to the database.
If it already exists, a pointer to that artifact will be returned.The parameters to theregisterArtifactfunction are meant fordocumentation, not as explicit directions to create the artifact from scratch.
In the future, this feature may be added to gem5art.Note: While creating new artifacts, warning messages showing that certain attributes (except hash and id) of two artifacts don't match (when artifact similarity is checked in the code) might appear. Users should make sure that they understand the reasons of any such warnings.Using artifacts from the databaseYou can create an artifact with just a UUID if it is already stored in the database.
The behavior will be the same as when creating an artifact that already exists.
All of the properties of the artifact will be populated from the database.ArtifactDBThe particular database used in this work isMongoDB.
We use MongoDB since it can easily store large files (e.g., disk images), is tightly integrated with Python throughpymongo, and has an interface that is flexible as the needs of gem5art changes.Currently, it's required to run a database to use gem5.
However, we are planning on changing this default to allow gem5art to be used standalone as well.gem5art allows you to connect to any database, but by default assumes there is a MongoDB instance running on the localhost atmongo://localhost:27017.
You can use the environment variableGEM5ART_DBto specify the default database to connect when running simple scripts.
Additionally, you can specify the location of the database when callinggetDBConnectionin your scripts.In case no database exists or a user want their own database, you can create a new database by creating a new directory and running the mongodb docker image.
See theMongoDB docker documentationor theMongoDB documentationfor more information.`dockerrun-p27017:27017-v<absolutepathtothecreateddirectory>:/data/db--namemongo-<sometag>-dmongo`This uses the officialMongoDB Docker imageto run the database at the default port on the localhost.
If the Docker container is killed, it can be restarted with the same command line and the database should be consistent.Connecting to an existing databaseBy default, gem5art will assume the database is running atmongodb://localhost:27017, which is MongoDB's default on the localhost.The environment variableGEM5ART_DBcan override this default.Otherwise, to programmatically set a database URI when using gem5art, you can pass a URI to thegetDatabaseConnectionfunction.Currently, gem5art only supports MongoDB database backends, but extending this to other databases should be straightforward.Searching the Databasegem5art provides a few convience functions for searching and accessing the database.
These functions can be found inartifact.common_queries.Specifically, we provide the following functions:getByName: Returns all objects mathchingnamein database.getDiskImages: Returns a generator of disk images (type = disk image).getLinuxBinaries: Returns a generator of Linux kernel binaries (type = kernel).getgem5Binaries: Returns a generator of gem5 binaries (type = gem5 binary).Downloading from the DatabaseYou can also download a file associated with an artifact using functions provided by gem5art. A good way to search and download items from the database is by using the Python interactive shell.
You can search the database with the functions provided by theartifactmodule (e.g.,getByName,getByType, etc.).
Then, once you've found the ID of the artifact you'd like to download, you can calldownloadFile.
See the example below.$python
Python3.6.8(default,Oct72019,12:59:55)[GCC8.3.0]onlinux
Type"help","copyright","credits"or"license"formoreinformation.
>>>fromgem5art.artifactimport*
>>>db=getDBConnection()>>>foriingetDiskImages(db,limit=2):print(i)...
ubuntuid:d4a54de8-3a1f-4d4d-9175-53c15e647afdtype:diskimagepath:disk-image/ubuntu-image/ubuntuinputs:packer:fe8ba737-ffd4-44fa-88b7-9cd072f82979,fs-x86-test:94092971-4277-4d38-9e4a-495a7119a5e5,m5:69dad8b1-48d0-43dd-a538-f3196a894804Ubuntuwithm5binaryinstalledandrootautologin
ubuntuid:c54b8805-48d6-425d-ac81-9b1badba206etype:diskimagepath:disk-image/ubuntu-image/ubuntuinputs:packer:fe8ba737-ffd4-44fa-88b7-9cd072f82979,fs-x86-test:5bfaab52-7d04-49f2-8fea-c5af8a7f34a8,m5:69dad8b1-48d0-43dd-a538-f3196a894804Ubuntuwithm5binaryinstalledandrootautologin
>>>foriingetLinuxBinaries(db,limit=2):print(i)...

vmlinux-5.2.3id:8cfd9fbe-24d0-40b5-897e-beca3df80dd2type:kernelpath:linux-stable/vmlinux-5.2.3inputs:fs-x86-test:94092971-4277-4d38-9e4a-495a7119a5e5,linux-stable:25feca9a-3642-458e-a179-f3705266b2feKernelbinaryfor5.2.3withsimpleconfigfile
vmlinux-5.2.3id:9721d8c9-dc41-49ba-ab5c-3ed169e24166type:kernelpath:linux-stable/vmlinux-5.2.3inputs:npb:85e6dd97-c946-4596-9b52-0bb145810d68,linux-stable:25feca9a-3642-458e-a179-f3705266b2feKernelbinaryfor5.2.3withsimpleconfigfile
>>>fromuuidimportUUID
>>>db.downloadFile(UUID('8cfd9fbe-24d0-40b5-897e-beca3df80dd2'),'linux-stable/vmlinux-5.2.3')For another example, assume there is a disk image namednpb(containingNAS ParallelBenchmarks) in your database and you want to download the disk image to your local directory. You can do the following to download the disk image:importgem5art.artifactdb=gem5art.artifact.getDBConnection()disks=gem5art.artifact.getByName(db,'npb')fordiskindisks:ifdisk.type=='disk image'anddisk.documentation=='npb disk image created on Nov 20':db.downloadFile(disk._id,'npb')Here, we assume that there can be multiple disk images/artifacts with the namenpband we are only interested in downloading the npb disk image with a particular documentation ('npb disk image created on Nov 20'). Also, note that there is not a single way to download files from the database (although they will eventually use the downloadFile function).The dual of thedownloadFilemethod used above isupload.Database schemaAlternative, you can use the pymongo Python module or the mongodb command line interface to interact with the database.
See theMongoDB documentationfor more information on how to query the MongoDB database.gem5art has two collections.artifact_database.artifactsstores all of the metadata for the artifacts andartifact_database.fsis aGridFSstore for all of the files.
The files in the GridFS use the same UUIDs as the Artifacts as their primary keys.You can list all of the details of all of the artifacts by running the following in Python.#!/usr/bin/env python3frompymongoimportMongoClientdb=MongoClient().artifact_databaseforiindb.artifacts.find():print(i)gem5art also provides a few methods to search the database for artifacts of a particular type or name. For example, to find all disk images in a database you can do the following:importgem5art.artifactdb=gem5art.artifact.getDBConnection('mongo://localhost')foriingem5art.artifact.getDiskImages(db):print(i)Other similar methods include:getLinuxBinaries(),getgem5Binaries()You can use getByName() method to search database for artifacts using the name attribute. For example, to search for gem5 named artifacts:importgem5art.artifactdb=gem5art.artifact.getDBConnection('mongo://localhost')foriingem5art.artifact.getByName(db,"gem5"):print(i)Artifacts API DocumentationArtifact Module
--------
.. automodule:: gem5art.artifact
    :members:

Artifact
--------
.. automodule:: gem5art.artifact.artifact
    :members:
    :undoc-members:

Artifact
--------
.. automodule:: gem5art.artifact.artifact.Artifact
    :members:
    :undoc-members:

Helper Functions for Common Queries
-----------------------------------
.. automodule:: gem5art.artifact.common_queries
    :members:
    :undoc-members:

AritifactDB
-----------
This is mostly internal.

.. automodule:: gem5art.artifact._artifactdb
    :members:
    :undoc-members: