Package: jupyter-on-golem
Jupyter on Golem - Python KernelAboutJupyter on Golem is a Python kernel, which integrates with JupyterLab (https://jupyterlab.readthedocs.io) and enables it to run your Python Notebooks using resources available on decentralized Golem Network (https://www.golem.network/). Providers of such resources are compensated for their work with GLM tokens.JupyterLab is an open-source project that is widely used by data scientists, analysts, researchers and developers. It allows you to create and share Notebooks - documents that combine code, equations, visualisations and narrative text. JupyterLab is part of Project Jupyter, umbrella project born from IPython Project (https://ipython.org/) and centered around providing tools for interactive computing with computational notebooks. Find more about Jupyter Project:https://jupyter.org/Simplified view on JupyterLab:Simplified view on Jupyter on Golem:Jupyter on Golem comes with following PyPI packages preinstallled:jupyter         - version 1.0.0pandas          - version 2.0.2tensorflow-cpu  - version 2.12.0matplotlib      - version 3.7.1Other packages can be installed on the Provider's host with%pip installcommand as described inCommandssection of this document.PrerequisitesJupyter on Golem needs some prerequsites. Please make sure you have the following requirements pre-installed:JupyterLab - version 4.0.2 or aboveIt most likely need no introduction. As stated on Jupyter Project webpage, JupyterLab is "highly extensible, feature-rich notebook authoring application and editing environment, and is a part of Project Jupyter, a large umbrella project centered around the goal of providing tools (and standards) for interactive computing with computational notebooks". To easy install JupoyterLab follow instructions provided on Jupyter Project webpage:https://jupyter.org/installTLDR install JupyterLab:pip install jupyterlabYagna - version 0.12.2 or aboveYagna is a default implementation of Golem Network Node. It will act as an intermediary for JupyterLab to be able to communicate with Golem Network and find provider to execute our Jupyter Notebooks. Yagna can work in two "modes": Requestor or Provider. Jupyter on Golem needs Yagna to run in Requestor mode. To easy install Yagna as a Requestor follow instructions provided on Golem Network webpage:https://handbook.golem.network/requestor-tutorials/flash-tutorial-of-requestor-development#easy-installationTLDR install Yagna:curl -sSf https://join.golem.network/as-requestor | bash -GLM tokensGolem is a Decentralized Network where Requestors (those who need computational resources) meet with Providers (those who rent such resources). Jupyter on Golem acts as a Requestor and consequently need a way to compensate Providers for their work. Golem utilize Blockchain technology to provide censorship resistant payments with GLM tokens (https://www.golem.network/glm).To sum it up You will need GLM tokens and some Network native tokens (ETH/Matic) for transaction fees to use Jupyter on Golem!However, We are aware that for some people Blockchain might be a little bit tricky at the start. That is why we have prepared two example scenarios of using Jupyter on Golem. One of them does not require from You any prior ownership of GLM tokens and is perfect for a first try!Testnet (Goerli) - For Blockchain newbies: This example will be performed on Testnet and as a result will not require from You any prior Blockchain knowledge. Testnet still needs tokens to pay providers but those will be given via%fundcommand. Nevertheless, You need to take into account that Testnet is for testing purposes. In other words it might be unstable from time to time and offers smaller amount of providers with less powerful equipment.Mainnet (Polygon) - For Blockchain pros: This one assumes that You are familliar with Blockchain and are fully capable of obtaining GLM + Matic tokens on your own. Mainnet is more stable and offers more powerful providers.Both scenarios are linked inExamplessection of this document.Increased number of open file descriptors - macOS only!This change is required formacOS users only!To apply it open terminal and perform following steps:Check Your current limit value of open file descriptors (most likely it will be 256) and write it down so You can go back to it if needed later:ulimit -nIncrease the limit to 65565:ulimit -n 65565Verify that the change took place:ulimit -nInstallationInstall Jupyter on Golem package from PyPI:pip install jupyter-on-golemAdd jupyter-on-golem to the list of known kernels:python -m jupyter_on_golem installUsageJupyter on Golem needs Yagna running on Your host to be able to talk with Golem Network. You can run it with single command (preferably in the separate Console Window):yagna service runWith Yagna up and running You can start JupyterLab in standard fashion:jupyter-labIf installation process was completed correctly then Golem kernel should be visible and available in JupyterLab web interface:Click on the upper Golem icon (right under the Notebook section) to open new Notebook. You should be able to run Jupyter on Golem magic commands like%help,which will display the most important information about Jupyter on Golem:%helpWhen You finish using Jupyter on Golem You can close JupyterLab (File -> Shut Down) and press Ctrl+C in Yagna terminal window to turn it off.CommandsBelow You will find the list of all Jupter on Golem magic commands which are required to interact with this tool.Magic commands is the concept of IPython (default kernel of JupyterLab). If you want to learn more then You should visit:https://ipython.readthedocs.io/en/stable/interactive/tutorial.html#magic-functions%helpShows Jupyter on Golem basic information and list of available magic commands with a short description.%statusShows the current status of Jupyter on Golem. With this command You can get following information:Your YAGNA node IDYour YAGNA wallet address- You should sent there GLM and MATIC tokens if You want to use Golem mainnet. By default wallet address is the same as node IDAmount of tokens (GLM,MATIC) hold by your wallet on MainnetAmount of tokens (tGLM,tETH) hold by your wallet on TestnetCurrent connection status- Disconnected or EstablishedName and Node ID of a Provider- for Established connections onlyResources of a Provider (RAM, DISK, CPU)- for Established connections onlyElapsed connection time- for Established connections only%fundRequests for Goerli testnet tokens: tGLM and tETH. You can only get Goerli testnet tokens this way.%fund goerli%budgetAllocates budget (GLM/tGLM) from funds in Your wallet for computation purposes on Jupter on Golem. It also sets the network to which you will connect when using%connectcommand! Using this command multiple times will each time overwrite the previous settings.To setup for Goerli Testnet and Allocate 2 tGLM:%budget goerli 1To setup for Polygon Mainnet and Allocate 2 tGLM:%budget polygon 2%connectInitialize a connection with a provider fulfilling specified requirements.following requirements can be specified:Minimal amount of RAM (GB)- memMinimal amount of disk space (GB)- diskMinimal amount of CPU cores- coresPlease take into account that sometimes it might take a litle bit longer to find a provider and setup him for work. To define how much You are willing to wait You can use timeout parameter. Default timeout is set to 10 minutes (timeout=10m)Sample connect command requesting for provider with at least 4GM of RAM, 10GB of Disk space and 2 CPU cores:%connect mem>4 disk>10 cores>2Sample connect command with timeout set to 15 minutes, requesting for provider with at least 4GM of RAM, 10GB of Disk space and 2 CPU cores:%connect mem>4 disk>10 cores>2 timeout=15m%disconnectDisconnects from the current provider and initializes the payment transaction. It also shows following information:Elapsed connection timeTotal cost of the connectionRemaining allocation%pip installInstalls specified pip package. It is imporant to do it as a magic command (with % prefix) for installation to be completed correctly!For instance, to add some colors to Your notebook You can install colorama package:%pip install coloramaExamplesWe have prepared some examples to help you play around with Jupyter on Golem and better feel what this solution is capable of. To run examples you need to import to Your JupyterLab following files:Testnet (Goerli) example notebookMainnet (Polygon) example notebookData Set (for both Testnet and Mainnet)Above examples are modification of "Linear Regression with a Real Dataset" Notebook from Machine Learning Crash Course created by Google:LINK. All exercises are made available under theApache License Version 2.0Data Set contains data drawn from the 1990 U.S. Census. It is also available as part of Machine Learning Crash Course created by Google:LINK. Tou can find data descriptionHERELimitationsJupyter on Golem is in its infancy stage. Consequently, it has many limitations You should be aware of. The most imporant are listed below:Currently Golem Network does not support easy way to utilize provider's GPU. Consequently, Jupyter on Golem is limited to utilize RAM, Disk space and CPU from the provider's resources.Data sets, results and other files need to be downloaded and uploaded via Requestor (i.e. Jupyter on Golem). In other words if You want to use some data sets from the Internet you need to download it locally first and then push into the provider host with%uploadcommand.Installing Python libraries is limited to PyPI with PIP.Using%pip installdoes not present live progress output. However, there is a spinner icon to let You know that installation process in ongoing.Outbound Internet access is limited to the most important services like above mentioned PyPI with PIP. Don't be surprised that other calls might not work.It might take a while to%connectto provider. Specifics depends on many factors, yet usually it takes something between 3 up to 10 minutes. You can also usetimeout=10mparemeter to define how much are you willing to wait. In general we recommend to use this time to get a tea or coffee ;)Currently only post-paid payments are supported. Single payment transaction is initialized after Requestor (Jupter on Golem) disconnects from the Provider.Golem Network is a free market where You receive multiple offers for a single demand. Multiple strategies can be applied to chose the offer. However, currently Jupyter on Golem supports only random strategy (i.e. choosing the random option among all offers received).FeedbackDid You try Jupyter on Golem? We would love to get to know your thoughts! Please give us your feedbackHERESupportIf you experience problems with Jupyter on Golem, You can try to get some help in the following places:jupyter-on-golem-discussionchannel on our Golem NetworkDiscord ServerGithub Repositoryof the projectKnown IssuesBelow You will find list of known issues and potential solutions:1. Unsuccessful funding on Goerli Testnet:Sometimes after using%fund goerlicommand, You will get"Funding failed"output. It might happen due to delay between requesting tETH and tGLM by Jupyter on Golem. In most cases, it should be enough to just wait a few minutes. Afterwards, type%statuscommand and You should be able to see that both tETH and tGLM have arrieved to Your wallet.Terms and ConditionsBy using Jupyter on Golem you agree to be bound by the terms described in Golem NetworkDisclaimer,User Interaction GuidelinesandPrivacy Policy.Legal Statements“Jupyter” and the Jupyter logos are trademarks or registered trademarks of NumFOCUS.