Package: s3contents
S3Contents - Jupyter Notebooks in S3A transparent, drop-in replacement for Jupyter standard filesystem-backed storage system.
With this implementation of aJupyter Contents Manageryou can save all your notebooks, files and directory structure directly to a
S3/GCS bucket on AWS/GCP or a self hosted S3 API compatible likeMinIO.Installationpipinstalls3contentsInstall with GCS dependencies:pipinstalls3contents[gcs]s3contents vs XWhile there are some implementations of an S3 Jupyter Content Manager such ass3nbors3drives3contents is the only one tested against new versions of Jupyter.
It also supports more authentication methods and Google Cloud Storage.This aims to be a fully tested implementation and it's based onPGContents.ConfigurationCreate ajupyter_notebook_config.pyfile in one of theJupyter config directoriesfor example:~/.jupyter/jupyter_notebook_config.py.Jupyter Notebook Classic: If you plan to use the Classic Jupyter Notebook
interface you need to changeServerApptoNotebookAppfor all the examples on this page.AWS S3froms3contentsimportS3ContentsManagerc=get_config()# Tell Jupyter to use S3ContentsManagerc.ServerApp.contents_manager_class=S3ContentsManagerc.S3ContentsManager.bucket="<S3 bucket name>"# Fix JupyterLab dialog issuesc.ServerApp.root_dir=""AuthenticationAdditionally you can configure multiple authentication methods:Access and secret keys:c.S3ContentsManager.access_key_id="<AWS Access Key ID / IAM Access Key ID>"c.S3ContentsManager.secret_access_key="<AWS Secret Access Key / IAM Secret Access Key>"Session token:c.S3ContentsManager.session_token="<AWS Session Token / IAM Session Token>"AWS EC2 role auth setupIt also possible to use IAM Role-based access to the S3 bucket from an Amazon EC2 instance or AWS resource.To do that just leave any authentication options (access_key_id,secret_access_key) to their default ofNoneand ensure that the EC2 instance has an IAM role which provides sufficient permissions (read and write) for the bucket.Optional settings# A prefix in the S3 buckets to use as the root of the Jupyter file systemc.S3ContentsManager.prefix="this/is/a/prefix/on/the/s3/bucket"# Server-Side Encryptionc.S3ContentsManager.sse="AES256"# Authentication signature versionc.S3ContentsManager.signature_version="s3v4"# See AWS key refreshc.S3ContentsManager.init_s3_hook=init_functionAWS key refreshThe optionalinit_s3_hookconfiguration can be used to enable AWS key rotation (describedhereandhere) as follows:fromaiobotocore.credentialsimportAioRefreshableCredentialsfromaiobotocore.sessionimportget_sessionfromconfigparserimportConfigParserfroms3contentsimportS3ContentsManagerdefrefresh_external_credentials():config=ConfigParser()config.read('/home/jovyan/.aws/credentials')return{"access_key":config['default']['aws_access_key_id'],"secret_key":config['default']['aws_secret_access_key'],"token":config['default']['aws_session_token'],"expiry_time":config['default']['aws_expiration']}asyncdefasync_refresh_credentials():returnrefresh_external_credentials()defmake_key_refresh_boto3(this_s3contents_instance):session_credentials=AioRefreshableCredentials.create_from_metadata(metadata=refresh_external_credentials(),refresh_using=async_refresh_credentials,method='custom-refreshing-key-file-reader')refresh_session=get_session()# from aibotocore.sessionrefresh_session._credentials=session_credentialsthis_s3contents_instance.boto3_session=refresh_session# Tell Jupyter to use S3ContentsManagerc.ServerApp.contents_manager_class=S3ContentsManagerc.S3ContentsManager.init_s3_hook=make_key_refresh_boto3MinIO playground exampleYou can test this using theplay.minio.io:9000playground:Just be sure to create the bucket first.froms3contentsimportS3ContentsManagerc=get_config()# Tell Jupyter to use S3ContentsManagerc.ServerApp.contents_manager_class=S3ContentsManagerc.S3ContentsManager.access_key_id="Q3AM3UQ867SPQQA43P2F"c.S3ContentsManager.secret_access_key="zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"c.S3ContentsManager.endpoint_url="https://play.minio.io:9000"c.S3ContentsManager.bucket="s3contents-demo"c.S3ContentsManager.prefix="notebooks/test"Access local filesTo access local file as well as remote files in S3 you can usehybridcontents.Install it:pipinstallhybridcontentsUse a configuration similar to this:froms3contentsimportS3ContentsManagerfromhybridcontentsimportHybridContentsManagerfromnotebook.services.contents.largefilemanagerimportLargeFileManagerc=get_config()c.ServerApp.contents_manager_class=HybridContentsManagerc.HybridContentsManager.manager_classes={# Associate the root directory with an S3ContentsManager.# This manager will receive all requests that don"t fall under any of the# other managers."":S3ContentsManager,# Associate /local_directory with a LargeFileManager."local_directory":LargeFileManager,}c.HybridContentsManager.manager_kwargs={# Args for root S3ContentsManager."":{"access_key_id":"<AWS Access Key ID / IAM Access Key ID>","secret_access_key":"<AWS Secret Access Key / IAM Secret Access Key>","bucket":"<S3 bucket name>",},# Args for the LargeFileManager mapped to /local_directory"local_directory":{"root_dir":"/Users/danielfrg/Downloads",},}GCP - Google Cloud StorageInstall the extra dependencies with:pipinstalls3contents[gcs]froms3contents.gcsimportGCSContentsManagerc=get_config(c.ServerApp.contents_manager_class=GCSContentsManagerc.GCSContentsManager.project="<your-project>"c.GCSContentsManager.token="~/.config/gcloud/application_default_credentials.json"c.GCSContentsManager.bucket="<GCP bucket name>"Note that the file~/.config/gcloud/application_default_credentials.jsonassumes
a POSIX system when you didgcloud init.Other configurationFile Save HooksIf you want to use pre/post file save hooks here are some examples.Apre_save_hookis written in the exact same way as normal, operating on the
file in local storage before committing it to the object store.defscrub_output_pre_save(model,**kwargs):"""Scrub output before saving notebooks"""# only run on notebooksifmodel["type"]!="notebook":return# only run on nbformat v4ifmodel["content"]["nbformat"]!=4:returnforcellinmodel["content"]["cells"]:ifcell["cell_type"]!="code":continuecell["outputs"]=[]cell["execution_count"]=Nonec.S3ContentsManager.pre_save_hook=scrub_output_pre_saveApost_save_hookinstead operates on the file in object storage,
because of this it is useful to use the file methods on thecontents_managerfor data manipulation.
In addition, one must use the following function signature (unique tos3contents):defmake_html_post_save(model,s3_path,contents_manager,**kwargs):"""Convert notebooks to HTML after saving via nbconvert"""fromnbconvertimportHTMLExporterifmodel["type"]!="notebook":returncontent,_format=contents_manager.fs.read(s3_path,format="text")my_notebook=nbformat.reads(content,as_version=4)html_exporter=HTMLExporter()html_exporter.template_name="classic"(body,resources)=html_exporter.from_notebook_node(my_notebook)base,ext=os.path.splitext(s3_path)contents_manager.fs.write(path=(base+".html"),content=body,format=_format)c.S3ContentsManager.post_save_hook=make_html_post_save