Package: bagit-create
bagit-create"BagIt Create" is a tool to export digital repository records in packages with a consistent format, according to theCERN Submission Information Package specification.Digital Repositories powered by Invenio v1, Invenio v3, Invenio RDM, CERN Open Data and Indico are supported, as well as GitLab repositories and locally found folders.Quick start:# Install
pip install bagit-create

# Create bag for CDS record 2728246
bic --recid 2728246 --source cdsTable of contentsInstallLXPLUSDevelopmentUsageExamplesOptionsFeaturesSupported sourcesURL parsingLight bagsConfigurationIndicoInvenio v1.xCERN SSOLocalCodiMDAdvanced usageModuleAccessing CERN firewalled websitesbibdocfileInstallPre-requisites:# On CentOSyuminstallgcckrb5-develpython3-develIf you just need to run BagIt Create from the command line:# Install from PyPipipinstallbagit-create# Check installed versionbic--version# Create bag for CDS record 2728246bic--recid2728246--sourcecdsLXPLUSBagIt-Create can be easily installed and used on LXPLUS (e.g. if you need access to mounted EOS folders):pip3installbagit-create--userCheck if.local/bin(where pip puts the executables) is in the path. If notexport PATH=$PATH:~/.local/bin.DevelopmentClone this repository and then install the package with the-eflag:# Clone the repositorygitclonehttps://gitlab.cern.ch/digitalmemory/bagit-createcdbagit-create# Create a virtual environment and activate itpython-mvenvenvsourceenv/bin/activate# Install the project in editable modepipinstall-e.# Check installed versionbic--version# You're done! Create a SIP for a CDS resource from its URL:bic--urlhttp://cds.cern.ch/record/2798105-v# Install additional packages for testingpipinstallpytestoais_utils# Run tests# Set an Indico API key, or expect some test to failexportINDICO_KEY=<YOUR_INDICO_KEY>exportGITLAB_KEY=<YOUR_GITLAB_KEY>
python-mpytestCode is formatted usingblackand linted withflake8. A VSCode settings file is provided for convenience.UsageYou usually just need to specify the location of the record you are trying to create a package for.You can do it by specifying the "Source" (seesupported sources) and the Record ID:bic--recid2728246--sourcecdsor passing an URL (currently only works with CDS, Zenodo and CERN Open Data links):bic --url http://cds.cern.ch/record/2665537ExamplesGitLab:bic --source gitlab --token <YOUR_TOKEN> --recid 104913 -vvCDS:# (Expect error) Removed resourcebic--recid1--sourcecds# (Expect error) Public resource but metadata requires authorisationbic--recid1000--sourcecds# Resource with a lot of large videos, light bagbic--recid1000571--sourcecds--dry-run# Resource with just a PDFbic--recid2728246--sourcecdsilcdoc:# ilcdoc #bic--sourceilcdoc--recid62959--verbose
bic--sourceilcdoc--recid34794--verboseZenodobic--recid3911261--sourcezenodo--verbose
bic--recid3974864--sourcezenodo--verboseIndicobic--recid1024767--sourceindicoCERN Open Databic--recid1--sourcecod--dry-run--verbose
bic--recid8884--sourcecod--dry-run--verbose--alternate-uri
bic--recid8884--sourcecod--dry-run--verbose
bic--recid5200--sourcecod--dry-run--verbose
bic--recid8888--sourcecod--dry-run--verbose

bic--recid10101--sourcecod--dry-run--verbose
bic--recid10102--sourcecod--dry-run--verbose
bic--recid10103--sourcecod--dry-run--verbose
bic--recid10104--sourcecod--dry-run--verbose
bic--recid10105--sourcecod--dry-run--verbose

bic--recid10101--sourcecod--verbose
bic--recid10102--sourcecod--verbose
bic--recid10103--sourcecod--verbose
bic--recid10104--sourcecod--verbose
bic--recid10105--sourcecod--verboseSome more advanced recipes can be found in theexamples/folder.Options--versionShowtheversionandexit.--recidTEXTRecordIDoftheresourcetheupstreamdigitalrepository.Requiredbyeverypipelinebutlocal.-s,--source[cds|ilcdoc|cod|zenodo|inveniordm|indico|local|ilcagenda]Selectsourcepipelinefromthesupportedones.-u,--urlTEXTProvideanURLfortheRecord[WorkswithCDS,OpenDataandZenodolinks]-d,--dry-runSkipdownloadsandcreatea`light`bag,withoutanypayload.-a,--alternate-uriUsealternativeuriinsteadofhttpsforfetch.txt(e.g.rootendpointsforCERNOpenDatainsteadofhttp).-v,--verboseEnablebasiclogging(verbose,'info'level).-vv,--very-verboseEnableverboselogging(veryverbose,'debug'level).-b,--bibdoc[ONLYforSupportedInveniov1pipelines]GetmetadataforaCDSrecordfromthebibdocfileutility(`/opt/cdsweb/bin/bibdocfile`mustbeavailableinthesystem).--bd-ssh-hostTEXT[ONLYforSupportedInveniov1pipelines]SpecifySSHhosttorunbibdocfile.Accessmustbepromptless.(Seedocumentationforusageandconfiguration).Bydefaultusesthelocalmachine.-t,--targetTEXTOutputfolderforthegeneratedSIP.Bydefaultusesthesamefolderthetoolisbeingexecutedfrom.-sp,--source-pathTEXT[LocalsourceONLY,required]Setpathofthelocalfoldertouseasasource.-u,--authorTEXT[LocalsourceONLY]SpecifytheAuthorofdata.-sbp,--source-base-pathTEXT[LocalsourceONLY]Specifyapartofthepathasrelevantforextractinganhierachy.-ic,--invcookieTEXT[Inveniov1.xONLY]UsecustomINVENIOSESSIONcookievaluetoauthenticate.Usefulforlocalaccounts.-ss,--skipssl[Inveniov1.xONLY]SkipSSLauthenticationinHTTPrequests.Usefulformisconfiguredordeprecatedinstances.-c,--certTEXT[Inveniov1.xONLY]Fullpathtothecertificatetousetoauthenticate.Don'tspecify extension, only the file name. A'.key' and a '.pem'willbeloaded.Readdocumentation(CERNSSOauthentication)tolearnmoreonhowtogenerateit.--helpShowthismessageandexit.FeaturesSupported sourcesNameSource IDURLPipelineCERN Document Servercdshttps://cds.cern.ch/Invenio v1.xILC Document Serverilcdochttp://ilcdoc.linearcollider.orgInvenio v1.xCERN Open Datacodhttps://opendata.cern.ch/CERN Open DataZenodozenodohttps://zenodo.org/Invenio v3.xCERN Indicoindicohttps://indico.cern.ch/Indico v3.0.xILC Agendailcagendahttps://agenda.linearcollider.org/Indico v3.0.xCodiMDcodimdhttps://codimd.web.cern.ch/CodiMDGitLabgitlabhttps://gitlab.cern.ch/GitLabAdditional configuration may be required (e.g. for restricted events).URL parsingInstead of passing Source + Record ID you can just use the record URL with the--urloption.Light bagsWith the--dry-runoption, BIC can create "light" bags skipping any payload download (i.e. attached files) but generating the same manifest (exposing upstream file locations and URLs), allowing the full bag to be "populated" in the future.ConfigurationSome pipelines require additional configuration (e.g. to authenticate).IndicoTo use any Indico pipeline you need an API Token. From your browser, login to the Indico instance, go to "Preferences" and then "API Token". Create new token, name can be anything. Select (at least)Everything (all methods)andClassic API (read only)as scopes. Note down the token and set theINDICO_KEYenvironement variable to it.exportINDICO_KEY=<INDICO_API_TOKEN>This will also allow you to run the tool forrestrictedevents you have access to.Invenio v1.xCERN SSOBIC can run in a "authenticated" mode for Invenio v1.x pipelines (e.g. CDS) by getting CERN SSO HTTP cookies through thecern-sso-pythontool.For this, you'll need to provide a Grid User certificate obtained from theCERN Certification Authorityof an account that has access to the desired restricted record.Once you downloaded your.p12certificate, you'll need to process the certificate files to remove passwords and separate the key and certificate:opensslpkcs12-clcerts-nokeys-inmyCert.p12-outmyCert.pem# A passphrase is required here (after the Import one)opensslpkcs12-nocerts-inmyCert.p12-outmyCert.tmp.key
opensslrsa-in~/private/myCert.tmp.key-outmyCert.keyWARNING: openssl rsa.. command removes the passphrase from the private key. Keep it in a secure location.Once you have yourmyCert.keyandmyCert.pemfiles, you can run BagIt-Create with the--certoption, providing the path to those files (without extension, as it is assumed that your certificate and key files have the same base name and are located in the same folder, and that the key has the file ending.keyand the certificate.pem). E.g.:bic--sourcecds--recid2748063--cert/home/avivace/Downloads/myCertWill make the tool look for "/home/avivace/Downloads/myCert.key" and "/home/avivace/Downloads/myCert.pem" and the pipeline will run authenticating every HTTP request with the obtained Cookies, producing a SIP of the desired restricted record.For more information, check thecern-sso-pythondocs.LocalTo authenticate with a local account (i.e. without CERN SSO), login on your Invenio v1.x instance with a browser and what yourINVENIOSESSIONcookie is set to.On Firefox, open the Developers tools, go in the "Storage" tab and select "Cookies", you should see anINVENIOSESSIONcookie. Copy its value and pass it to BagIt Create with the--tokenoption:bic--sourcecds--recid2748063--token<INVENIOSESSION_value_here>CodiMDTo create packages out of CodiMD documents, go tohttps://codimd.web.cern.ch/, authenticate and after the redirect to the main page open your browser developer tools (CTRL+SHIFT+I), go to the "Storage" tab and under cookies copy the value of theconnect.sidcookie.The "Record ID" for CodiMD document is the part of the url that follows the main domain address (e.g. inhttps://codimd.web.cern.ch/KabpdG3TTHKOsig2lq8tnw#the recid isKabpdG3TTHKOsig2lq8tnw)bic--sourcecodimd--recidvgGgOxGQU--token<connect.sid_value_here>Dump full historyA small script is included in this repository inexamples/codimd_history.pywhich will dump your entire CodiMD "history" (the same history you see on the homepage), creating a bag for each document.Set the CODIMD_SESSION env variable to the value of theconnect.sidcookie before running the script:CODIMD_SESSION=<connect.sid_value_here>pythonexamples/codimd_history.pyGitLabTo export projects from CERN GitLab you'll need to provide a Personal access token of an user that has at least a "Maintainer" role on the target repository. This token will also be used to clone the repository.By default, only files from the default branch will be indexed in the metadata and in the SIP manifest. The exported package will however also contain a full copy of the git repository, including every available branch.A package created using the "dry run" flag will not contain the repository copy.Advanced usageModuleBIC can easily be run inside other Python scripts. Just import it and use theprocessmethod with the same parameters you can pass to the CLI.E.g., this snippet creates SIP packages for CDS resources from ID 2728246 to 27282700.importbagit_createforiinrange(2728246,27282700):result=bagit_create.main.process(source="cds",recid=i,loglevel=3)ifresult["status"]==0:print("Success")else:print("Error")Accessing CERN firewalled websitesIf the upstream source you're trying to access is firewalled, you can set up a SOCKS5 proxy via a SSH tunnel through LXPLUS and then runbicthrough it with tools likeproxychainsortsocks. E.g.:Bring up the SSH tunnel:ssh-D1337-q-N-f-Clxplus.cern.chThe proxy will be up atsocks5://localhost:1337. After having installedtsocks, edit the its configuration file (/etc/tsocks.conf) as follows:[...]server=localhostserver_type=5server_port=1337[...]Now, just runbicas documented here but prependtsocksto the command:tsocksbic--recid1024767--sourceindico-vvbibdocfileThebibdocfilecommand line utility can be used to get metadata for CDS, exposing internal file paths and hashes normally not available through the CDS API.If the executable is available in the path (i.e. you can run/opt/cdsweb/bin/bibdocfile) just append--bibdoc:bic--recid2751237--sourcecds--bibdoc-vIf this is not the case, you can pass a--bd-ssh-hostparameter specifying the name of an SSH configured connection pointing to a machine able to run the command for you. Be aware that your machine must be able to establish such connection without any user interaction (the script will runssh <THE_PROVIDED_SSH_HOST> bibdocfile ..args).Since in a normal CERN scenario this can't be possible due to required ProxyJumps/OTP authentication steps, you can use theControlMasterfeature of any recent version of OpenSSH, allowing to reuse sockets for connecting:Add an entry in~/.ssh/configto set up the SSH connection to the remote machine able to runbibdocfilefor you in the following way:Host<SSH_NAME>User<YOUR_USER>Hostname<HOSTNAME.cern.ch>ProxyJump<LXPLUS_or_AIADM>ControlMasterautoControlPath~/.ssh/control:%h:%p:%rThen, runssh <SSH_NAME>in a shell, authenticate and keep it open. OpenSSH will now reuse this socket everytime you run<SSH_NAME>, allowing BagItCreate tool to runbibdocfileover this ssh connection for you, if you pass thebd-ssh-hostparameter:bic--recid2751237--sourcecds--bibdoc--bd-ssh-host=<SSH_NAME>-v