Package: emr-notebooks-magics
EMR Notebooks iPython MagicsThis repository contains iPython magics that can be used in Amazon EMR Notebooks.Table of ContentsInstallationUsageSecurityLicenseInstallationInstalling Dependencies%mount_workspace_dirmagic mounts the Workspace usingS3-FUSEorGoofys.Installing S3-FUSEAdd the following lines to your cluster bootstrap action script.#!/bin/sh

sudo amazon-linux-extras install epel -y
sudo yum install s3fs-fuse -yInstalling GoofysAdd the following lines to your cluster bootstrap action script.#!/bin/sh

sudo wget https://github.com/kahing/goofys/releases/latest/download/goofys -P /usr/bin/
sudo chmod ugo+x /usr/bin/goofysInstalling iPython magicsUsing EMR Step.EMR step script#!/bin/sh
sudo -u emr-notebook /mnt/notebook-env/bin/pip install emr-notebooks-magicsFrom Jupyter Notebook%pip install emr-notebooks-magicsThe magics are loaded using kernel startup script. If you install magics from Jupyter Notebook, you will need to restart the kernel before using the magic.Note: EMR-notebook-magics cannot be installed through bootstrap actions as JEG and Notebook environments are installed after the bootstrap.Usage%generate_s3_download_urlmagic generates presigned url for S3 objects so that it can be downloaded from the Jupyter Notebook.
Refer%generate_s3_download_url?for help.Generate download url for a S3 object specifying full S3 path.%generate_s3_download_url s3://my_bucket/path/to/s3/objectGenerate download url for a file in the Workspace specifying path relative to Workspace root.%generate_s3_download_url relative/path/to/workspace/file%mount_workspace_dirmagic mounts Workspace files on the EMR cluster instance using FUSE based filesystem.
Refer%mount_workspace_dir?for help.Mount the entire Workspace onto EMR cluster instance.%mount_workspace_dir .Mount a sub-directorymydirectoryand adduse_cachemount option of S3-FUSE%mount_workspace_dir mydirectory --params use_cache=/tmp/Mount a sub-directorymydirectoryand addcheap,regionmount option for Goofys.%mount_workspace_dir mydirectory --use goofys --params cheap,region=us-east-1%execute_notebookmagic executes another notebook in the background.
Consider executing long-running notebooks in the background to ensure that the output is continuously captured
even in case of a local network disruption.  The output of the executed cells are incrementally captured in a
new notebook with the same name as the executed notebook. The output notebook is placed inside a separate folder
within the Workspace. Additional permissions are required forEMR-EC2 instance roleto execute
this magic. Refer %execute_notebook? for help.Execute a notebook in the Workspace%execute_notebook <relative-file-path>Execute a notebook specific cluster id and notebook service role%execute_notebook <notebook_name>.ipynb --cluster-id <emr-cluster-id> --service-role <emr-notebook-service-role>:exclamation:  WarningsWhen the write access is enabled, any changes made to the mount directory are applied to the S3 Workspace. These changes are irreversible, please enable S3 versioning to your S3 Workspace as a pre-caution.Once the Workspace is mounted on the EMR cluster, it can be accessed from all EMR Notebooks in your account that can attach to that cluster.When you install S3-FUSE or Goofys, its your responsibility to keep those package up to date for new patches. Since Goofys is not managed by any package managers, take necessary steps to upgrade Goofys binaries.SecuritySeeCONTRIBUTINGfor more information.LicenseThis project is licensed under the Apache-2.0 License.