Package: clmutils
colab miscellaneous utilsMiscellaneous utils mainly intended for use in colabInstallationpipinstallclmutils# clm: colab-misc# pip install clmutils -U to updateOr clone the repohttps://github.com/ffreemt/colab-misc-utils.gitin Colab/jupyter:!git clone https://github.com/ffreemt/colab-misc-utils.git
%cd colab-misc-utils
!pip install -r requirements.txtUsage:  with a setting file dotenv/.envsetup_gitfromclmutilsimportsetup_git,Settingsconfig=Settings()setup_git(user_name=config.user_name,user_email=config.user_email,priv_key=config.gh_key)Do the usualgit pull, amend codes andgit pushstuff.For more onConnecting to GitHub with SSH, refer tohttps://docs.github.com/cn/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-sshsetup_ssh_tunnelRun these lines in Colabfromclmutilsimportsetup_ssh_tunnel,Settingsconfig=Settings()setup_ssh_tunnel(remote_host=config.remote_host,remote_user=config.remote_user,priv_key=config.cl_key,remote_pubkey=config.remote_pubkey)Append the value ofcl_key_pubindotenv/.env(below) in remote computer's~/.ssh/authorized_keys.In the remote computer:sshcolabclmutils.Settingwill look fordotenvor.envin/content/drive/MyDriveif google drive is momunted; otherwise it looks for.envin the current dir and parents dir.dotenv/.envbasically contains the necessary information to setup git for github or ssh tunnel:For git:gh_key: private keyuser_name/user_email;For reverse ssh tunnel:cl_key/cl_key_pub: private key/public key of Colabremote_pubkey: public_key of the remote computerremote_user: login namedotenv/.envis read byclmutils.Settinas followsfromclmutilsimportSettingsconfig=Settings()# config = Setting(_env_file=file_path)configwill then have attributes:config.gh_key, user_name, user_email, cl_key, cl_key_pub, remote_pubkey, remote_user.
If a particular attribute is not assigned a value indotenv/.env, it defaults to empty string ("").dotenvor.envhas the following info and format (lines starting with a#are comments):# for gituser_email="...@......"user_name="......"gh_key="-----BEGIN EC PRIVATE KEY-----...............................................................9...............................................................JK9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==-----END EC PRIVATE KEY-----"# for ssh tunnelremote_host="168.138.222.163"remote_user="ubuntu"# colab's private key, to be put in ~/.ssh/id_ed25519# or as specified in ~/.ssh/configcl_key="-----BEGIN OPENSSH PRIVATE KEY-----.....................................................................WQyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQAAAJDlRQSF5UUEhQAAAAtzc2gtZWQyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQ.....................................................................fDQRG5jrz/VCKKt2uu2eBAAAACWZvci1jb2xhYgECAwQ=-----END OPENSSH PRIVATE KEY-----"# to be put in remote computer's ~/.ssh/autorized_keyscl_key_pub="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIKVka3/2JsB32jG5Zx+5KOfDQRG5jrz/VCKKt2uu2eB colab-key"# to be put in colab's ~/.ssh/authorized_keysremote_pubkey="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE60sowZ4M0MA5nTGIH1RN54zERTuWSddFKqyeWZzaKv for-colab"Usage: withtout dotenv/.envSet upgithubwith ssh usingclmutils.setup_gitFor manually setting up github with ssh, refer tohttps://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-sshAssume you configure git as follows:gitconfig--globaluser.emailyour-email-address
gitconfig--globaluser.nameyour-github-usernameWithclmutils, you'd do:fromclmutilsimportsetup_git`user_email="your-email-address"user_name="your-github-username"gh_key=\"""-----BEGIN EC PRIVATE KEY-----MH.............................................................9AwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJK9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==-----END EC PRIVATE KEY-----""".strip()+"\n"setup_git(user_email=user_email,user_name=user_name,priv_key=gh_key)You then upload thepublic keyforgh_keytohttps://github.com/settings/keys.Refer to Step 2https://support.cloudways.com/using-git-command-line-ssh/for how to generate a private/public key pair. You can also use clmutils.gen_keypair to do that in Python.Alternatively, set upgithubwith ssh in 4 stepsWrite a private key to~/.ssh/gh-keyfromclmutilsimportcreate_filegh_key=\"""-----BEGIN EC PRIVATE KEY-----MH.............................................................9AwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJK9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==-----END EC PRIVATE KEY-----""".strip()+"\n"# Do not remove .strip() + "\n"# the private key is very picky about formatcreate_file(gh_key,dest="~/.ssh/gh-key")Set upgithub.comconfig forgit pushfromclmutilsimportappend_contentconfig_github_entry=\"""Host github.comHostName github.comUser gitIdentityFile ~/.ssh/gh-key"""append_content(config_github_entry,dest="~/.ssh/config")Verify that everything is OK, from a cell!ssh -o StrictHostKeyChecking=no -T git@github.comIf you see something similar toHiyour-name!You'vesuccessfullyauthenticated,butGitHubdoesnotprovideshellaccess.you are good to go.git config --globalYou can now set upgit config globalfrom a cell, e.g.!git config --global user.email your-email-address
!git config --global user.name your-github-username
# !ssh-keyscan github.com >> ~/.ssh/known_hostsYou are ready to clone your own repo, run your app and generate new data, update the repo and push togithub.Utils planned:white_check_mark:setup_gitsets upgitforgithub.com:white_check_mark:reverse_ssh_tunnelsets up a reverse ssh tunnel to a remote host with via autosshAuxiliary utils:white_check_mark:create_filecreates a file with given mode, e.g. for.ssh/id_rsaorIdentityFilein.ssh/config:white_check_mark:apppend_contentappends some content to a file, e.g., for appended a public key to.ssh/authorized_keys:white_check_mark:chmod600chmodof a file:white_check_mark:gen_keypairgenerates private/public key pair.:white_check_mark:run_cmdwrapssubprocess.check_output:white_check_mark:run_cmd1wrapssubprocess.Popen:white_check_mark:check_runningshows running processes with names containings a certain patter; based onpsutil.Proceess()'scmdlineandstatus`Demo: notebooks in Colabgit push from Colab: one line (setup_git())!pip install clmutils
from clmutils import setup_git
gh_key = """..."""
user_name = "..."
user_email = "..."
setup_git(setup_git(user_email=user_email, user_name=user_name, priv_key=gh_key)git pushfrom Colab in several steps(in Chinese, shouldn't be too difficult to follow without knowing any Chinese though, just click through :smiley:)Reverse ssh tunnel for ssh to Colab VMin English (I may provide a Chinese version later)