Package: g3po
g3poAssorted utility CLI to work with Gen3Required:Python 3.6, 3.7pip install g3po

g3po --help
g3po versionGeneralEnvironment VariableYou can overrideGEN3_URLenvironment variable to switch to different data commonsGEN3_URL=https://caninedc.org/ g3po index list | jqOr, exportGEN3_URLin your shell environment:export GEN3_URL=https://caninedc.org/
g3po index get 0614e421-0cd6-4f93-ad1d-5f9354928bd2 | jqAnd unset theGEN3_URLenv var:unset GEN3_URLCredentialsSome sub-commands are required privilege access e.g.g3po index delete --help
Usage: g3po index delete [OPTIONS] GUID

Options:
  --cred TEXT  Path to credentials.json i.e. API key from your profile
  --help       Show this message and exit.All sub-commands has optional--credoption. If this is not provided, it assumes to load from~/.gen3/credentials.jsondirectory.You can provide--credpath tocredentials.jsonfileg3po index delete --cred /path/credentials.json bd59f90a-286d-4688-96a6-777a6f1df79dOutput FormatYou can pipejqwithg3pooutput for pretty JSON formatg3po index list | jqGet index by GUID/DIDg3po index get 1543974f-93e7-4f67-85ac-802e19ec11e8 | jqGenerate UUIDg3po uuid
g3po uuid --help
g3po uuid --count 10 | jqData Dictionary SubcommandYou can convert/compile YAML schemas into a JSON as follows:g3po dd convert local/yaml_schemas/ --out local/my_dd.jsone.g.git clone https://github.com/umccr/umccr-dictionary.git
cd umccr-dictionary
g3po dd convert dictionary/umccr/ --out /tmp/umccr.json
g3po dd convert dictionary/gdc/ --out /tmp/gdc.json
g3po dd convert dictionary/anvil/ --out /tmp/anvil.jsonIndex Subcommandg3po index --help
g3po index health
g3po index stats
g3po index listCreating Blank IndexHelpg3po index blank create --help
Usage: g3po index blank create [OPTIONS]

Options:
  --uploader TEXT  Uploading user within system, e.g. user.name@g3po.org
                   [required]

  --authz TEXT     Authz resources split by comma, e.g.
                   /programs/program1/projects/P1  [required]

  --filename TEXT  File name, e.g. test.bam
  --cred TEXT      Path to credentials.json downloaded from your profile
  --help           Show this message and exit.You can create ablank index recordas follows:g3po index blank create --uploader user.name@g3po.org --authz /programs/program1/projects/P1 | jq
{
  "baseid": "93906e3e-6a1a-4e65-813b-dcec0763ae7b",
  "did": "835a367f-4f7d-4996-a562-e23f9d57cdeb",
  "rev": "6c75a7e8"
}Here, make sure--uploaderoption set to your login username for your Gen3 common site.And, also make sure--authzresource ACL has properly applied to your account in Fenceuser.yamlconfig.Blank index record has nothing yet. You can delete it back if you like so:g3po index delete 835a367f-4f7d-4996-a562-e23f9d57cdebUpdating Blank IndexHelpg3po index blank update --help
Usage: g3po index blank update [OPTIONS]

Options:
  --guid TEXT        Blank GUID/DID to update e.g.
                     1543974f-93e7-4f67-85ac-802e19ec11e8  [required]

  --rev TEXT         Current revision e.g. 7bd043f3  [required]
  --hash_type TEXT   Hash type e.g. md5  [required]
  --hash_value TEXT  Hash value e.g. ab167e49d25b488939b1ede42752458b
                     [required]

  --size INTEGER     File size in bytes, e.g. 1024  [required]
  --authz TEXT       Authz resources split by comma, e.g.
                     /programs/program1/projects/P1  [required]

  --urls TEXT        Resource URLs split by comma, e.g.
                     s3://bucket/1543974f-93e7-4f67-85ac-802e19ec11e8/test.txt

  --cred TEXT        Path to credentials.json downloaded from your profile
  --help             Show this message and exit.You can update a blank index record with hashes, size, etc... as follows:g3po index blank update \
  --guid bd59f90a-286d-4688-96a6-777a6f1df79d \
  --rev 5f8e7f9d \
  --hash_type md5 \
  --hash_value d0df0d078def0a5d7eb8ed6eb1e06099 \
  --size 7149868 \
  --urls s3://g3po-gen3-dev/bd59f90a-286d-4688-96a6-777a6f1df79d/somatic-PASS.vcf.gz \
  --authz /programs/program1/projects/P1 \
  | jq

{
  "baseid": "811a7786-16ff-46d4-b9a2-1d647bdebfb9",
  "did": "bd59f90a-286d-4688-96a6-777a6f1df79d",
  "rev": "c203ed4d"
}Manifest IndexingHelpg3po index manifest --help
Usage: g3po index manifest [OPTIONS]

Options:
  --cred TEXT       Path to credentials.json downloaded from your profile
  --tsv TEXT        Path to manifest.tsv file
  --thread INTEGER  Number of thread, default 8
  --output TEXT     Output file
  --help            Show this message and exit.Generate Manifest Templateg3po index template

tree .
.
└── manifest.tsvManifest Format:guid- leave blank if you like Gen3 to generate GUID/DID.Otherwise, generate UUID like so:g3po uuid --count 10 | jqoruuid v4md5- usemd5sumto produce hashes for your file resourcesize- determine your file resource size in bytes.authz- comma separated list of associated DD and resource authz ACL path that configured in Fenceuser.yaml.file_name- File nameurls- comma separated list of the file resource URLsValidate Manifest Formatg3po index validate
validating "manifest.tsv" manifest
mapped manifest column "md5" to "MD5Validator" class instance
mapped manifest column "urls" to "URLValidator" class instance
mapped manifest column "size" to "SizeValidator" class instance
mapped manifest column "authz" to "AuthzValidator" class instance
finished validating "manifest.tsv" manifest, no errors were found
manifest.tsv is valid!Run Indexingg3po index manifest

tree .
.
├── manifest.tsv
└── manifest_output_1601392690.tsv

less manifest_output_1601392690.tsvUser Subcommandg3po user --helpUser YAML from LDAPBefore executing, the environment variableG3PO_LDAP_PASSWORDmust be set to the
password of the LDAP user (see full options below).g3po user ldap --user-in-yaml sample/user.in.yamlThis will transform the givenuser.in.yamlusing information retrieved from the
corresponding LDAP server. The transformed YAML will be output to stdout.Full options as follows:export G3PO_LDAP_PASSWORD=<option_ldap-user_password_here>

g3po user ldap \
  --user-in-yaml sample/user.in.yaml \
  --ldap-server "ldaps://ldap-test.cilogon.org" \
  --ldap-user "uid=readonly_user,ou=system,o=UMCCR,o=CO,dc=biocommons,dc=org,dc=au" \
  --ldap-search-base "ou=people,o=UMCCR,o=CO,dc=biocommons,dc=org,dc=au" > user.yamlValidate User YAMLg3po user validate --help
g3po user validate user.yaml
g3po user validate user1.yaml user2.yamlNOTE: this is same-as or alias togen3user validate user.yamlDevelopmentActivate virtual environmentAnd install pip dependenciespip install -e '.[dev,tests]'
which g3po
g3po version