Package: numerai-cli
numerai-cliWelcome to the Numerai CLI for theNumerai Tournament.
This README is designed to have EVERYTHING you need to setup and maintain a Numerai Compute Heavy node.This CLI runs on your local computer to configure a Numerai Prediction Node in the cloud.
This solution is architected to cost less than $5/mo on average, but actual costs may vary.
It has been tested on MacOS/OSX, Windows 8/10, and Ubuntu 18/20,
but should theoretically work anywhere that Docker and Python 3 are available.ContentsGetting StartedList of CommandsUpgradingUpgrading from 0.1/0.2 to 0.3.0BeyondUninstallingTroubleshooting and FeedbackPythonDockerMacOS and Windows 10LinuxOlder PCs: Docker ToolboxAzureCommon ErrorsBilling AlertsPrediction Node ArchitecturePython ExampleRLang ExampleThe DockerfileCloud ComponentsSpecial ThanksGetting StartedTo use this tool you need:Numerai API keysPaid cloud provider accountPythonDockerSign up a Numerai Account, get your Numerai API Keys, and your first Model:Sign up athttps://numer.ai/signupand log in to your new accountGo tohttps://numer.ai/account> "Your API keys" section > click "Add"Name your key and check all boxes under "Scope this key will have"Enter your password and click "Confirm"Copy your secret public key and secret key somewhere safeChoose your cloud provider and follow the corresponding guide:AWS Setup GuideAzure Setup GuideGCP Setup GuideInstall Docker and Python for your Operating System (if you encounter errors or your
OS is not supported, please seeTroubleshooting and Feedback):Mac Terminal (cmd + space, typeterminal, selectterminal.app):curlhttps://raw.githubusercontent.com/numerai/numerai-cli/master/scripts/setup-mac.sh|bashUbuntu 18/20 Terminal (ctrl + alt + t):sudoaptupdate&&sudoaptinstall-ylibcurl4curl&&sudocurlhttps://raw.githubusercontent.com/numerai/numerai-cli/master/scripts/setup-ubu.sh|sudobashWindows 10 Command Prompt (windows key, typecmd, select Command Prompt):powershell-command"$Script = Invoke-WebRequest 'https://raw.githubusercontent.com/numerai/numerai-cli/master/scripts/setup-win10.ps1'; $ScriptBlock = [ScriptBlock]::Create($Script.Content); Invoke-Command -ScriptBlock $ScriptBlock"After the setup script confirms Python and Docker, installnumerai-clivia:pip3install--upgradenumerai-cli--userNOTES:This command will also work to update to new versions of the package in the future.If you are using python venv then drop the --user option.
If you don't know what that is, disregard this note.Run these commands on your personal computer (not a temporary instance)
to get an example node running in minutes:For AWS run:numeraisetup--providerawsand enter your user's keys when prompted.For Azure users run:numeraisetup--providerazureand enter your application's keys when prompted.For GCP users run:numeraisetup--providergcpand enter the complete path to your downloaded key when prompted.numerainodeconfig--exampletournament-python3
numerainodedeploy
numerainodetestIf you want to use larger instances to generate your predictions first runnumerai list-constantsto list the vCPU/mem presets available, then you can configure a node to use one of the presets via:numerainodeconfig-smem-lgYour compute node is now setup and ready to run! When you make changes to your code or re-train your model,
simply deploy and test your node again:numerainodedeploy
numerainodetestNOTES:These commands have stored configuration files in$USER_HOME/.numerai/. DO NOT LOSE THIS FILE!
or else you will have to manually delete every cloud resource by hand.The example node trains a model in the cloud, which is bad. You should train locally, pickle the
trained model, deploy your node, then unpickle your model to do the live predictionsThe default example doesnotmake stake changes; please refer to thenumerapi docsfor the methods you must call to do this.You can view resources and logs in the AWS Console (region us-east-1) for yourECS Clusterandother resourcesList of CommandsUse the--helpoption on any command or sub-command to get a full description of it:numerai
numerai--help
numerai[command]--help
numerai[command][sub-command]--helpEach command or sub-command takes its own options, for example if you want to copy the
numerai signals example and configure a node for a signals model with large memory
requirements you'd use something like this (replacing [MODEL NAME] with the relevant
signals model name):numerainode-m[MODELNAME]-sconfig-smem-lg-esignals-python3Here, thenodecommand takes a model name with-mand a flag-sto detect if it's
a signals model or numerai model. Theconfigsub-command also takes a-soption to
specify the size of the node to configure.UpgradingUpgrading numerai-cli will always require you to update the package itself using pip:pipinstall--upgradenumerai-cli--user
numeraiupgrade0.1/0.2 to 0.3CLI 0.3 uses a new configuration format that is incompatible with versions 0.1 and 0.2,
but a command to migrate you configuration is provided for you. Run this in the directory
you rannumerai setupfrom the previous version:numeraiupgrade0.3 to 0.4CLI 0.4 introduces a new provider option (Microsoft Azure) and moves the default aws
terraform into a subdirectory. You'll need to runupgrade:numeraiupgradeIf you want to use azure, follow thesetup guide for azurethen run:numeraisetup--providerazure
numerainodeconfig--providerazure0.4 to 1.0CLI 1.0 is a big change for AWS, you'll need to:update the permissions for the role you created in theAWS Setup Guideupdate the terraform withnumerai setupreplace your aws nodes by running:numerai node  -m <model_name> destroy --preserve-node-confignumerai node -m <model_name> confignumerai node -m <model_name> deployBeyondSome updates will make changes to configuration files used by Numerai CLI. These will
require you to re-run some commands to upgrade your nodes to the newest versions:numerai setupwill copy over changes to files in the$HOME/.numeraidirectorynumerai node configwill apply those changes to a nodeUninstallingnumeraiuninstallTroubleshooting and FeedbackPlease review this entire section and check github issues before asking questions.
If you've exhausted this document, then join us on DiscordIf you still cannot find a solution or answer, please join us onDiscordand include the following information with your issue/message:The commands you ran that caused the error (even previous commands)Version information from running:pip3 show numerai-clipython -Vdocker -vSystem Information from runningMac:system_profiler SPSoftwareDataType && system_profiler SPHardwareDataTypeLinux:lsb_release -a && uname -aWindows:powershell -command "Get-ComputerInfo"PythonIf the environment setup script fails to install Python3 for you, report the error to Numerai
and then install it manually with one of the following options:Download Python3 directlyinstall it fromyour system's package managerUsecondato install and manage python for youDockerIf the environment setup script fails to install Docker for you, report the error to Numerai
then read the following to get a working installation on your machine. For PCs, you may need toactivate virtualization in your BIOSbefore installing.MacOS and Windows 10Just installDocker Desktopto get it running.
You should also increase the RAM allocated to the VM:Open the Docker Desktop GUI thenClick Gear in top right cornerSelect Resources > Advanced in left sidebarUse slider to allocate more memory (leave a few gigs for your OS and background applications, otherwise your computer might crash)LinuxCheck here for instructions for your distribution:https://docs.docker.com/engine/install/Older PCs: Docker ToolboxIf your machine is older and/or doesn't have Hyper-V enabled, then you will have to follow these steps to install docker toolbox on your machine:Install Oracle VirtualBoxfor your Operating SystemRestart your computerInstall Docker ToolboxRestart your computerAfter it's installed, open the "Docker QuickStart Terminal" and run the following to increase its RAM:docker-machinermdefault
docker-machinecreate-dvirtualbox--virtualbox-cpu-count=2--virtualbox-memory=4096--virtualbox-disk-size=50000default
docker-machinerestartdefaultNOTE: your code must live somewhere under your User directory (ie. C:\Users\USER_NAME\ANY_FOLDER). This is a restriction of docker toolbox not sharing paths correctly otherwise.AzureIf you just made your Azure account there's a chance your account provisioning could take some time, potentially up to 24 hours.When configuring your node for the first time the numerai-cli may hang as it tries to provision infrastructure in your account. If runningnumerai node config --provider azureshows no log output for more than 5 minutes, your account is likely in the stuck provisioning state. While we investigate this issue, the best course of action is to wait until the following day and run the command again as there is no way to skip this Azure provisioning step.Common ErrorsError:
subprocess.CalledProcessError:Command'docker run --rm -it -v /home/jason/tmp/.numerai:/opt/plan -w /opt/plan hashicorp/terraform:light init'returnednon-zeroexitstatus127.


Reason:
Dockerisnotinstalled.

Solutions:
Ifyou're certain that docker is installed, make sure that your user can execute docker, ie. try to run `docker ps`.If that'stheissue,thendependingonyoursystem,youcandothefollowing:

-Windows/OSX-MakesuretheDockerDesktopisrunningandfinishedbootingup.Itcantakeafewminutestobecompletelyready.Whenclickingonthedockertrayicon,itshouldsay"Docker Desktop is Running".-Ifyou're using Docker Toolbox on Windows, then make sure you'veopenedthe"Docker Quickstart Terminal".

-Linux-Run`sudousermod-aGdocker$USER`-Thenrebootorlogout/loginforthistotakeeffect.Error:
docker:Errorresponsefromdaemon:Drivehasnotbeenshared

Solutions:
-Youneedto[shareyourdrive](https://docs.docker.com/docker-for-windows/#shared-drives).Error:
numerai:commandnotfound

Solutions:
-osx/linux:Tryandrun`~/.local/bin/numerai`-Windows:`%LOCALAPPDATA%\Programs\Python\Python37-32\Scripts\numerai.exe`-Alternatively,exitthenre-openyourterminal/commandprompt.Error:
errorcallingsts:GetCallerIdentity:InvalidClientTokenId:Thesecuritytokenincludedintherequestisinvalid.
...
Command'docker run -e "AWS_ACCESS_KEY_ID=..." -e "AWS_SECRET_ACCESS_KEY=..." --rm -it -v /home/jason/tmp/.numerai:/opt/plan -w /opt/planhashicorp/terraform:light apply -auto-approve'returnednon-zeroexitstatus1.

Solutions:
-Run`numeraiconfigure`tore-writeyourAPIKeysError:
ERRORnumerapi.base_api:Errorreceivedfromyourwebhookserver:{"tasks":[],"failures":[{"reason":"The requested CPU configuration is above your limit"}]}Solution:1.Gotothe[QuotaDashboardforEC2](console.aws.amazon.com/servicequotas/home/services/ec2/quotas)2.Searchfor"On-Demand",thiswilllistallinstancetypesandtheirlimitsforyouraccount.3.Clickthebubblenextto"Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances"4.click"Request quota increase"inthetopright5.InputahighervaluethanthecurrentlyappliedquotaandfinishtherequestIf after AWS increases your quota, you still get this error, try again 1-2 more timesYou may have to complete a quota request for other types of instances too depending on how resource intensive your setup isBilling AlertsThere's no automated way to setup billing alerts, so you'll need toconfigure one manually.
We estimate costs to be less than $5 per month unless your compute takes more than 12 hours a week,
but increasing the RAM/CPU will increase your costs.Prediction Node ArchitectureA Prediction Node represents a cloud-based model that Numerai can trigger for predictions; it is designed to be reliable, resource efficient, and easy to configure and debug. Prediction Nodes use a few important components like aDockerfile, aTrigger, aContainer, and aCompute Cluster, all of which can be created using one of the following examples.Python Examplenumerai-python3
├──Dockerfile
├──.dockerignore
├──predict.py
├──requirements.txt
└──train.pyDockerfile: Used duringnumerai node deployto build a Docker image that's used to run your code in the cloud. It copies all files in its directory, installs Python requirements for requirements.txt, and runspython predict.pyby default..dockerignore: This file uses regex to match files that should not be included in the Docker image.train.py: This is an extra entry point specifically for training, it's used when runningnumerai node test --local --command "python train.py"requirements.txt: Defines python packages required to run the code.predict.py: Gets run by default locally and in the cloud when runningnumerai testwithout the--command|-coption.RLang Examplenumerai-rlang
├──Dockerfile
├──.dockerignore
├──install_packages.R
└──main.RDockerfile: Used duringnumerai node deployto build a Docker image that's used to run your code in the cloud. It copies all files in its directory, installs Rlang requirements from install_packages.R, and runs main.R by default..dockerignore: This file uses regex to match files that should not be included in the Docker image.install_packages.R: Installs dependencies necessary for running the example.main.R: Ran by default locally and in the cloud and when runningnumerai testwithout the--command|-coption.The DockerfileThis is the most important component of deploying a Prediction Node. It is a program (much like a bash script), that packages up your code as animage; this image contains everything your code needs to run in the cloud. The most typical case of a Dockerfile is demonstrated inNumerai Examples, if you're not sure how to use a Dockerfile, first copy an example withnumerai copy-example, then read the documentation in the Dockerfile to learn the basics.These files are very flexible, the default Dockerfile will just copy everything in whatever directory it is in, but this can be customized if you'd like to share code between models. For example, if you have a python project setup like so:numerai_models
├──common
├────__init__.py
├────data.py
├────setup.py
├──model_1
├────Dockerfile
├────.dockerignore
├────predict.py
├────requirements.txt
└────train.pyWherecommonis an installable python package you want to use in multiple models, you can add this line to model_1/Dockerfile:RUN pip install ../common/. Finally, runnumerai node deployfrom thenumerai_modelsdirectory to install the package in the image, making it available to your model code.If you want to learn more about how to customize this file [checkout the Dockerfile reference] (https://docs.docker.com/engine/reference/builder/).Cloud ComponentsThe CLI usesTerraformto provision cloud resources. Each component and the related cloud resource(s) are listed below. The links will take you to the AWS console where you can monitor any of these resources for a given node; just visit the link and select the resource with the same name as the node you want to monitor (further directions are given for each resource below).Trigger: A small function that schedules a "task" on yourCompute Cluster. This "task" handles pulling the image that was created by theDockerfileand running it as aContaineron yourCompute Cluster. This is handled by two resources:API Gateway:
Hosts the webhook (HTTP endpoint) that Numerai calls to trigger your nodes.
After clicking the link and selecting the resource, use the left sidebar to access metrics and logging.Lambda:
Schedules your compute job when you call your Webhook URL.
After clicking the link and selecting the resource, use the "Monitor" tab below the "Function Overview" section.Container: The thing that actually contains and runs your code on a computer provisioned by theCompute Cluster. The--size(or-s) flag on thenumerai node configsets the CPU and Memory limits for aContainer. This is stored in one place:ECR (Elastic Container Repository):
Used for storing docker images. This is the location to whichnumerai docker deploywill push your image.
There is not much monitoring here, but you can view your images and when they were uploaded.Compute Cluster: A handler that accepts scheduled "tasks" and spins up and down computers to runContainers. This is handled by ECS:ECS (Elastic Container Service):
This is where your containers will actually run and where you'll want to look if your containers don't seem to be scheduled/running.
After clicking the link, you'll be able to scroll and monitor the top-level metrics of each cluster.
After selecting a specific cluster, you can use the various tabs to view different components of the cluster (tasks are the runnable jobs
that the Lambda schedules, instances are the computers the tasks run on, and metrics will show cluster-wide information)Special ThanksThanks touuazedfor their work onnumerapiThanks tohellnofor starting the Signalsticker mapThanks to tit_BTCQASH (numerai profileandtwitter profile) for debugging the environment setup process on Windows 8Thanks toeses-wkfor implementing Azure support