Package: onecodex
One Codex API - Python Client Library and CLICommand line interface (CLI) and Python client library for interacting with the One Codex v1 API (API docs).MAINTAINERS:@clausmith,@boydgreenfieldInstallationThis package provides 3 major pieces of functionality: (1) a core Python client library; (2) a simple CLI for interacting with the One Codex platform that uses that core library; and (3) optional extensions to the client library, which offers many features aimed at advanced users and provides functionality for use in interactive notebook environments (e.g., IPython notebooks).Python 3.8 or later is required.Python 2 is no longer supported.Basic installationThe CLI (and core Python library) may be simply installed usingpip. To download a minimal installation (#1 and #2), simply run:pipinstallonecodexInstallation with optional extensionsTo also download the optional extensions to the client library, and all of their dependencies, run:pipinstall'onecodex[all]'Using the CLILogging inThe CLI supports authentication using either your One Codex API key or your One Codex username and password.
To log in using your username and password:onecodexloginThis command will save a credentials file at~/.onecodex, which will then automatically be used for authentication the next time the CLI or Python client library are used (OS X/Linux only). You can clear this file and remove your API key from your machine withonecodex logout.In a shared environment, we recommend directly using your One Codex API key, rather than logging in and storing it in a credentials file. To use API key authentication, simply pass your key as an argument to theonecodexcommand:onecodex--api-key=YOUR_API_KEYsamplesYour API key can be found on theOne Codex settings pageand should be 32 character string. You may also generate a new API key on the settings page in the web application.Note: Because your API key provides access to all of the samples and metadata in your account, you should immediately reset your key on the website if it is ever accidentally revealed or saved (e.g., checked into a GitHub repository).Uploading filesThe CLI supports uploading FASTA or FASTQ files (optionally gzip compressed) via theuploadcommand.onecodexuploadbacterial_reads_file.fq.gzMultiple files can be uploaded in a single command as well:onecodexuploadfile1.fq.gzfile2.fq.gz...You can also upload files using the Python client library:uploaded_sample1=ocx.Samples.upload("/path/to/file.fastq")# Or upload a tuple of paired end filesuploaded_sample2=ocx.Samples.upload(("/path/to/R1.fastq","/path/to/R2.fastq"))Which returns aSamplesresource (as of0.5.0). Samples can be associated with tags, metadata, and projects at upload timing using those respective keyword arguments:# Note format must match the schema defined for our API, with arbitrary# metadata allowed as a single-level dictionary in the `custom` field.# See https://developer.onecodex.com/reference#the-metadata-resource for details.metadata={"platform":"Illumina NovaSeq 6000","date_collected":"2019-04-14T00:51:54.832048+00:00","external_sample_id":"my-lims-ID-or-similar","custom":{"my-string-field":"A most interesting sample...","my-boolean-field":True,"my-number-field-1":1,"my-number-field-2":2.0,}}Uploads can be made in parallel using Python threads (or multiple processes), e.g.:importconcurrent.futuresuploaded_samples=[]withconcurrent.futures.ThreadPoolExecutor(max_workers=4)asexecutor:futures={executor.submit(ocx.Samples.upload,file)forfileinLIST_OF_FILES}forfutureinconcurrent.futures.as_completed(futures):try:uploaded_samples.append(future.result())exceptExceptionase:print("An execption occurred during your upload:{}".format(e))ResourcesThe CLI supports retrieving your One Codex samples and analyses. The following resources may be queried:Your samples (Samples)Sample metadata (Metadata)Analyses, which include several subtypes with additional functionality and fields:Classifications, which are basic metagenomic classification results for your samplesPanels, which arein silicopanels for particular genes or other functional markers (example on One Codex)Jobs, which provide information on the name, version, and type of analysis which was performed for a givenAnalysesSimply invoke theonecodexcommand, using one of the above resource names as a subcommand (all lowercase). For example:# fetch all your samplesonecodexsamples# fetch a list of panels based on their idsonecodexpanels0123456789abcdef0987654321fdecbaUsing the Python client libraryInitializationTo load the API, use the following import:fromonecodex.apiimportApiInstantiate an API client either by passing your API key or automatically fetching your credentials from~/.onecodexif you've previously calledonecodex login.fromonecodex.apiimportApi# Instantiate a One Codex API object, will attempt to get credentials from ~/.onecodexocx=Api()# Instantiate an API object, manually specifying an API keyocx=Api(api_key="YOUR_API_KEY_HERE")ResourcesResources are exposed as attributes on the API object. You can fetch a resource directly by its ID or you can fetch it using the query interface. Currently you can access resources using eitherget()orwhere(). If you need help finding the ID for a sample, its identifier is part of its url on our webpage: e.g. for an analysis athttps://app.onecodex.com/analysis/public/1d9491c5c31345b6, the ID is1d9491c5c31345b6. IDs are all short unique identifiers, consisting of 16 hexadecimal characters (0-9a-f).sample_analysis=ocx.Classifications.get("1d9491c5c31345b6")# Fetch an individual classificationsample_analysis.results()# Returns classification results as JSON objectsample_analysis.table()# Returns a pandas dataframeIn addition to methods on individual instances of a given resource (e.g., aSampleor anAnalysis), the library also provides methods for aggregating sets of samples or analyses:all_completed_analyses=ocx.Classifications.where(complete=True)all_completed_analyses.to_otu()# Returns classification results as JSON objectall_completed_analyses.to_df()# Returns a pandas dataframeDevelopmentEnvironment SetupBefore developing,gitandpythonversion >=3.8 are needed. We recommend usingpyenvfor Python version management.To download the client library from GitHub:gitclonehttps://github.com/onecodex/onecodex.gitcdonecodex/To set up the project, first create a virtual environment and then install dependencies:# If you are on a M1 Macbook, run the line below, adjusting the version as neededexportHDF5_DIR=/opt/homebrew/Cellar/hdf5/1.12.1_1/

virtualenvvenvsourcevenv/bin/activate
pipinstall-e'.[all,testing,reports]'# -e specifies development mode so that the package doesn't have to be reinstalled after code editsTest are run via the Makefile. Note this may take awhile at first because of installing dependencies:makelint
maketestWe usepre-commitfor automated linting usingblack,flake8, and various whitespace and newline formatters during development.Writing Unit TestsWe usepytestas our unit testing framework. Tests should be able to run without an internet connection, and One Codex API calls must be mocked. We useresponsesto mock API responses.Tip:Any API calls that do not have a matching mock will raise an error. You can figure out which API calls need to be mocked by writing a test, running it, and inspecting the error message to see which route(s) are missing.Warning:Mocked URLswithouta query string will ignore query strings in any matching requests. If the mocked URLincludesa query string, it will be used when matching requests.FixturesThese pytest fixtures may be helpful when writing unit tests:ocx: this is a mockedApiobject that uses theapi/v1One Codex API schema.ocx_experimental: this is a mockedApiobject that uses theapi/v1_experimentalOne Codex API schema. Use this to test experimental features.api_data: this mocks some API data forv1andv1_experimentalAPIs.Mocking API DataAPI data, including schemas, are stored intests/data/api/:tests/data/api
├── v1  # the API version
│   ├── ...
│   ├── analyses
│   │   └── index.json  # payload for accessing GET::api/v1/analyses. Will also be used to mock each resource instance, e.g. GET::api/v1/analyses/<uuid>
│   ├── classifications
│   │   ├── 0f4ee4ecb3a3412f
│   │   │   └── results
│   │   │       └── index.json  # payload for accessing GET::api/v1/classifications/0f4ee4ecb3a3412f/results
│   │   └── index.json  # payload for accessing GET::api/v1/classifications. Instance routes are also auto-mocked
│   ├── ...
│   ├── schema
│   │   ├── index.json  # payload for accessing GET::api/v1/schema
│   │   └── index_all.json  # payload for accessing GET::api/v1/schema?expand=all
│   └── ...
└── v1_experimental
    └── ...The directory structure mirrors the One Codex API. For example:The payload for API routeapi/v1/classificationsis stored attests/data/api/v1/classifications/index.json.API routeapi/v1/classifications/0f4ee4ecb3a3412f/resultshas its payload stored attests/data/api/v1/classifications/0f4ee4ecb3a3412f/results/index.json.For thev1_experimentalAPI, store things underapi/v1_experimental/.This idea can be extended to arbitrary nesting/depths within the API.Note:If the payload is large, you can gzip it and name itindex.json.gz.A resource's instance list payload (e.g.api/v1/analysesgives you a list of analyses) is used to auto-mock each resource instance (e.g.api/v1/analyses/<uuid>). You don't need to create anindex.jsonfor each instance.Mocking API SchemasAPI schemas work similarly to regular API data, but with a couple of special rules:Each API schema directorymusthaveindex.jsonandindex_all.json.index.jsoncontains the payload for e.g.api/v1/schema, andindex_all.jsoncontains the payload for e.g.api/v1/schema?expand=all.If the schema requires it, you can optionally define resource-specific schemas by including<resource-name>.jsonin theschemadirectory (e.g.assemblies.jsonforapi/v1_experimental/assemblies/schema).conftest.pyAPI data is loaded intests/conftest.py. If you need to mock API calls in a way that's not supported by this framework, you can add custom mocked calls inconftest.py.Things that arenotsupported by mocking intests/data/api/:Non-GET requests (e.g. DELETE)Query parameters (with the exception ofschema?expand=all)Jupyter Notebook Custom ExportersWe also package custom Jupyter notebooknbconvertexporters. These can be tested with the following snippets and the providedexample.ipynbreport.OurOneCodexHTMLExporter:ONE_CODEX_REPORT_FILENAME=example.htmljupyternbconvert--execute--toonecodex_html--ExecutePreprocessor.timeout=-1--output="$ONE_CODEX_REPORT_FILENAME"--output-dir="."notebook_examples/example.ipynb&&openexample.htmlAnd using theOneCodexPDFExporter:ONE_CODEX_REPORT_FILENAME=example.pdfjupyternbconvert--execute--toonecodex_pdf--ExecutePreprocessor.timeout=-1--output="$ONE_CODEX_REPORT_FILENAME"--output-dir="."notebook_examples/example.ipynb&&openexample.pdfNote these require the installation of severalnpmpackages for headless rendering:npm install vega-lite vega-cli canvas. Seealtair-saverfor more details.