TITLE: Steps for setup of AWS organization, S3 data storage, and EC2 computing for using Python notebooks
AUTHORS: Daniel J. Pollak, Gautam Chawla, Andrey Andreev, David A. Prober

[DESCRIPTION]
With the oncoming age of big data, biologists are encountering more use cases for cloud-based computing to streamline data processing and storage. Unfortunately, cloud platforms are difficult to learn, and there are few resources geared towards biologists for demystifying them. We have developed a guide for experimental biologists to set up cloud processing on Amazon Web Services to cheaply outsource data processing and storage. Here we provide a guide on setting up a computing environment in the cloud and showcase examples of using Python and Julia programming languages. We present example calcium imaging data in the zebrafish brain and corresponding analysis using suite2p software. Tools for management of users and budgets are discussed in the protocol. Following this guide should help researchers even with limited programming experience to get started or move existing coding infrastructure into the cloud environment.


[STEPS]
SECTION: Setting organization and budget management
1. Setting up a Root account
SECTION: Setting organization and budget management
1.1. Create “root” account for your organization, using Business account type
  

SECTION: Setting organization and budget management
1.2. If you are using GMail account to manage other services in your lab, for example labname@gmail.com, you can use labname+aws@gmail.com to register account with AWS (so-called “gmail + trick”)
SECTION: Setting organization and budget management
1.3. You will have to enter credit card information to register root account. Consult your department if you want to avoid using personal credit card
SECTION: Setting organization and budget management
2. Research credits  can be applied to the account of the organization. Contact entity that issued the credits (most likely your IT department)
SECTION: Setting organization and budget management
3. There are two options to allow users be part of the organization.
External account can be added to organization, but it might be better to create accounts within organization interface:

 

SECTION: Setting organization and budget management
4. After an account has been created, lab member should receive email and use the “reset password” function to set up new password.
SECTION: Setting organization and budget management
5. Each member of the organization can have an individual budget set up. Monthly cost budget is a good starting point to manage costs on a per-account basis. Because sometimes cost can skyrocket accidentally due to misconfigured software (by the users) budget alerts and limits will provide safeguard against such overruns

 
 

SECTION: EC2: Starting up a computing instance
7. We consulted two tutorials to set up computing instances:
By Justin Bois, the AWS setup and usage lesson in his Caltech BE/Bi 103b class "Statistical Inference in the Biological Sciences" https://bebi103b.github.io/lessons/08/aws_setup.html
By Chris Albon: Run Project Jupyter Notebooks On Amazon EC2
SECTION: EC2: Starting up a computing instance
8. Find a new instance image (operating system) from the AWS marketplace, either Amazon Linux 2, or Ubuntu 18.04 or 20.04.
We recommend using Amazon Linux 2
SECTION: EC2: Starting up a computing instance
9. On the Launch page, in the dropdown Choose Action menu, choose Launch through EC2
  
SECTION: EC2: Starting up a computing instance
10. This will open a Launch wizard. In the Launch Instance wizard, you will be brought through a 5 part launch sequence.
SECTION: EC2: Starting up a computing instance
10.2. Choose Instance Type. 
This will configure your virtual machine "hardware" such as memory size and CPU.
Some relevant parameters here are memory size (RAM), number of cores, and internet speed. For most applications, RAM should be at least 32 GB, the number of cores should be at least 8, and the internet should be very fast, especially because we will be streaming tens or even hundreds of GB of data from S3 to EC2.
  
SECTION: EC2: Starting up a computing instance
10.3. Configure instance details 
Accept defaults
SECTION: EC2: Starting up a computing instance
10.4. Add Storage
This local storage is fast but will be destroyed after you terminate instance. To start, give your instance around 3x the size of your current dataset. This provides very fast "local" storage to the instance (compared to slower S3 storage)
SECTION: EC2: Starting up a computing instance
10.5. Add Tags
This is optional for better organization of instances
SECTION: EC2: Starting up a computing instance
10.6. Configure Security group
Most of the time you want port 22 (ssh) to be accessible for remote connections. Jupyter Notebooks server by default runs at port 8888, so that one should also be open.

Following Justin Bois’s tutorial, your security rules should look like this.
 
SECTION: EC2: Connecting to and using your instance
11. If you have just created the instance, it will start automatically. 
If the instance has been "Stopped", start your instance. Process should take less than few minutes.

NB: at the end of the work with the instance, you can either Stop or Terminate instance.
Stopped instance will have all its memory saved in S3 storage (and you will be billed for every GB of stored data according to S3 prices). Terminated instance will have its memory completely destroyed, and you will lose any information saved in the instance.
If you don't stop the instance, you will be billed for the time it is Running, regardless of whether you are actually using it for any computations or not. 
We recommend stopping instances at the end of work. Check with AWS pricing and your budget to estimate cost of storing instance data.
 

SECTION: EC2: Connecting to and using your instance
13. Using web-based terminal: Select connect, which will open the terminal in another tab

 


SECTION: S3: Uploading data via web-interface
23. To start, create a Bucket for your experimental data. It is important to block public access to the bucket
  

SECTION: S3: Setting up access for S3 bucket from AWS/EC2 instance
26. For each user, add their canonical ID to the bucket’s access list (need to allow both list and write permissions). That will allow these users to access bucket from Python interface
   
  

SECTION: Creating AMI (Amazon Machine Image) from configured instance
33. It is useful to save your configured machine image for later. Creating Image will save all packages installed, all data, and all private/sensitive information too. 

SECTION: Starting Jupyter lab
19. 
You are mostly finished with dependencies! You’ll add more as necessary, but these are the most important. We will treat this section as though you have freshly started your EC2 instance.
SECTION: Installing dependencies
15. Instance is a virtual computer, and you should keep it organized like any other computer. For example, you might need to get code using Git or download from the internet. Cloned or downloaded git repositories go in the git folder, and all other downloads go in the Downloads folder. To create these folders use commands:

 

SECTION: EC2: Starting up a computing instance
10.7. Review Instance Launch
Select Launch. This will open the Key Pair popup.
This is going to give you a “key”. Put it somewhere safe, meaning a location that is not synched to the Internet (Dropbox or Google Drive). The key will grant full access to the instance.
Click launch instance. It usually takes less than two minutes for instance to start.
SECTION: EC2: Connecting to and using your instance
12. Click Instance ID to view the instance summary page and select connect.

SECTION: Installing dependencies
16. Per Chris Albon's tutorial, install conda as follows:
SECTION: Installing dependencies
16.1. On the downloads page for https://anaconda.com, right click on the download link for Linux, and select “Copy link address”. 

 

If your local machine computer runs Windows or Mac instead of Linux, you will likely be prompted with a link for your particular operating system (OS). Take care not to copy that link. The link should end in .sh, not .exe or .pkg and look something like this: 

 


To install use commands:


 


Follow the prompts on the screen to complete the installation
SECTION: Installing dependencies
16.2. Follow the suite2p install instructions here: https://github.com/MouseLand/suite2p. 
The first step is to download a YAML file, which is not necessarily clear on Linux. To download it, we will use wget again: 


 

SECTION: Installing dependencies
16.3. Activate your new suite2p environment:
 

SECTION: Installing dependencies
16.4. Install jupyter lab:

 

SECTION: Installing dependencies
16.5. If you skipped Anaconda installation, use 


 
and

 


SECTION: Installing dependencies
16.6. Install smart_open for s3:

 

Make sure you install this library for s3 specifically, as specified here.
SECTION: Installing dependencies
17. If your EC2 instance’s OS is not Amazon Linux 2, install aws cli: 

 

SECTION: Installing dependencies
18. Install Julia
SECTION: Installing dependencies
18.1. From your Downloads directory:
 

 

SECTION: Installing dependencies
18.2. Install IJulia. This will allow you to run interactive julia environment inside Jupyter notebooks.
First, start Julia

 
And then install IJulia

 

SECTION: Installing dependencies
18.3. Within Julia interactive shell install necessary packages to work with TIFF images:

 
Note that we are installing dependencies in a slightly different way than how we installed IJulia. These methods are equivalent.
SECTION: Starting Jupyter lab
20. Start conda environment:
 

SECTION: Starting Jupyter lab
21. Open a persistent jupyter lab session: 

 
nohup command starts the jupyter server in the background, not tied to your connection to EC2 instance via ssh. The log created by jupyter server will be written to nohup.out file

Several parameters are specified here so that:
Jupyter lab will be accessible from your browser window (--ip=0.0.0.0)
The memory size limit is increased (--NotebookApp.max_buffer_size=75368709120). For applications that use a lot of memory like suite2p, Kilosort, Caiman, etc Python needs to make really big arrays, and the default memory size for jupyter lab (but not for the AWS instance) is too small. For work with 20GB datasets we use ~75 GB RAM
SECTION: Starting Jupyter lab
22. Open the nohup.out file to get the token.

 
Copy the string following "lab?token="
  
Use this token to login into the Python notebook:
 

Alternatively, you might need to start up a persistent server. A persistent server will not depend on the connection between your personal computer and the EC2 instance.

SECTION: Starting Jupyter lab
22.1. The server will start a process. To find its ID, run in terminal:

To stop Jupyter notebook server (or lab) copy the process number and kill it:
 
 

 

To stop Jupyter notebook server (or lab) copy the process number and kill it:

 



SECTION: Accessing files from Python using boto3
32. We have just shown how to confirm that the S3 bucket is visible to your EC2 instance. To run full data processing pipelines with suite2p and other such programs, we need to go one step further, and download files into our local filesystem. Let’s go through some Python code to see how it works.

[1]:
 

[2]:
 
[Out]:
 
[3]:
 
[4]:
 
[5]:
 
[Out]:
 

SECTION: Creating AMI (Amazon Machine Image) from configured instance
33.1. Go to Actions → Image and Templates → Create Image

 

SECTION: Creating AMI (Amazon Machine Image) from configured instance
33.2. Pick an appropriate name (for example “conda-image” for version with full installation of Python and Conda) and click Create Image
NB: the image will be created with all the data saved on the “local” disk. So if you have instance with 100GB of data saved, data will be added to the image as well

Note that images are stored on S3 storage service, and you will be charged for storing every image

Now you can launch an identical instance of your customized AMI by selecting your image from “My AMIs” when spinning up a new EC2 instance.
 

SECTION: S3: Setting up access for S3 bucket from AWS/EC2 instance
26.1. Canonical ID can be found in AWS Web Interface -> My Security Credentials -> Canonical User ID (AWS manual)
SECTION: S3: Setting up access for S3 bucket from AWS/EC2 instance
27. To allow downloading data from web browser interface, create a custom rule. It gives users who did not initially upload the data access to it through S3’s web interface at https://s3.console.aws.amazon.com/
  
Here the "Principal" is the user that gets permission to read data from the bucket specified in "Resource" field
SECTION: S3: Setting up access for S3 bucket from AWS/EC2 instance
28. Each user will have to create custom key pair for your account under security Credentials → Access keys.
This will allow you to create Access Key ID and Secret Access Key.
  


SECTION: S3: Setting up access for S3 bucket from AWS/EC2 instance
29. Connect to the EC2 instance using EC2 Connect function or ssh. Run aws configure to configure the instance.
Make sure to enter the correct zone ID.


 


SECTION: S3: Setting up access for S3 bucket from AWS/EC2 instance
30. While logged into the instance, run python to launch the Python interpreter
SECTION: S3: Setting up access for S3 bucket from AWS/EC2 instance
31. Test connection to bucket via the boto3 library:

 
This should print all items in the bucket

SECTION: S3: Uploading data via web-interface
24. You can use web interface to drag-and-drop data to upload it into the bucket or into a folder within bucket
  
Expected speed of upload is around 10MB/s.
Largest file size allowed through web-interface is 160GB.
SECTION: S3: Uploading data via web-interface
25. You can also use command-line interface to upload data in more automated way or upload larger files
SECTION: Setting organization and budget management
6. It is simple to check the spending by individual account using Cost Explorer in the organization Billing Dashboard.
You can select account (Linked Account). It is important to also pick correct Charge Type (most commonly you can use Credit)

  

SECTION: EC2: Starting up a computing instance
10.1. Chose AMI
Amazon Machine Image (AMI) is the blueprint of the operating system (OS). It can be Linux, FreeBSD, or even Windows. We recommend Amazon Linux 2 for this guide. Later on in this guide we discuss creating your own, personalized, private Amazon Machine Images.
SECTION: EC2: Connecting to and using your instance
14. Using terminal / ssh: