Package: teamconnector
Team Connector CLIOverviewteamconnectoris a command-line tool for interacting with various cloud storage and remote server platforms. It provides a simple and unified interface for managing files and directories across different platforms, making it easier to work with data in a distributed environment.InstallationBefore installingteamconnector, make sure you create a Conda environment for your project.
If you have our team Makefile, you can use themake create-envcommand to create a Conda environment.To installteamconnector, you can use pip:pip install teamconnectorUser-specific parametersTo use the Team Connector CLI, you need to set the following environment variables:REMOTE_USER: The username for the remote cluster filesystem.REMOTE_HOST: The hostname for the remote cluster filesystem.GOOGLE_ROOT: The root directory for your Google Drive.MY_DRIVE: The path to your Google Drive's "My Drive" folder.SHARED_DRIVE: The path to your Google Drive's "Shared Drives" folder.ONE_DRIVE: The path to your OneDrive folder.SLACK_USER_ID: Your Slack user ID, which can be found in your Slack profile settings.SLACK_BOT_TOKEN: The bot token for your Slack app, which can be obtained from the Slack API website.The following environment variables can be set in your bash profile (~/.bash_profileon Mac;~/.bashrcin Unix).However, I suggest you create an.envfile in your root directory (~) with the following:HOME=/Users/[Your_Username]
REMOTE_USER=[Your_Remote_Username]
REMOTE_HOST=[Your_Remote_Host]
GOOGLE_ROOT="/Users/[Your_Username]/Library/CloudStorage/[Your_GoogleDrive_Account]"
MY_DRIVE="/Users/[Your_Username]/Library/CloudStorage/[Your_GoogleDrive_Account]/My Drive"
SHARED_DRIVE="/Users/[Your_Username]/Library/CloudStorage/[Your_GoogleDrive_Account]/Shared Drives"
ONE_DRIVE="/Users/[Your_Username]/Library/CloudStorage/[Your_OneDrive_Account]"
SLACK_USER_ID=[Your_Slack_User_ID]
SLACK_BOT_TOKEN=[Your_Slack_Bot_Token]Within your conda environment, runtcinit ~/.envto load all the parameters into your conda environment.Project-specific parametersTo configure your Conda environment, you'll need to set a few environment variables:CLOUD_ROOT: This is the base name of your Google bucket. For example, if your Google bucket URL isgs://gpc_array, thenCLOUD_ROOTshould be set togs://gpc_array.PROJECT_ROOT: This is the absolute path to your local project folder. Replace<user>with your username. For example, if your project is in the/User/<user>/projects/gpc_arrayfolder, setPROJECT_ROOTaccordingly.REMOTE_DIR: The path to the directory on the remote cluster filesystem where your project files are located. For example, if your project is in the/gpfs/commons/groups/[Your_Group_Name]/users/[Your_User_Name]/[Your_Project_Name], set theREMOTE_DIRaccordingly.REMOTE_DATADIR: The path to the directory on the remote cluster filesystem where your data files are located. For example, if the project is in/gpfs/commons/groups/[Your_Group_Name]/projects/[Your_Project_Name], set theREMOTE_DATADIRaccordingly.You can set these variables using the following command:condaenvconfigvarssetCLOUD_ROOT=gs://gpc_arrayPROJECT_ROOT=`pwd`REMOTE_DATADIR=/gpfs/commons/groups/singh_lab/projects/gpc_array
condaenvconfigvarssetREMOTE_DIR=/gpfs/commons/groups/[Your_Group_Name]/users/[Your_User_Name]/[Your_Project_Name]If you are on the cluster,condaenvconfigvarssetREMOTE_DIR=`pwd`Alternatively, you can create an.envfile in your project repository folder with the followingREMOTE_DIR=/gpfs/commons/groups/[Your_Group_Name]/users/[Your_User_Name]/[Your_Project_Name]
REMOTE_DATADIR=/gpfs/commons/groups/singh_lab/projects/gpc_arrayand run:tcinit.envLinking the /data directory to a separate location (SLURM cluster-specific)On the NYGC cluster, the data directory is shared and located in a separate location than the user folder, where the repositories are. In this case, we should run the following command to link thedata/folder within your repository with the separate data directory:tcinit -lortcinit -l <datadir>TheREMOTE_DATADIRenvironment variable needs to be set if the<datadir>argument is not provided.Additional Configuration fordatatrackerIf you're usingdatatrackerand find yourself in more complex scenarios, you'll also need to set:TRACKER_PATH: This is the absolute path to thedb.jsonfile within your project. You can dynamically set this to thedb.jsonfile in your project folder using the$PROJECT_ROOTvariable you've already set.Execute the following command to setTRACKER_PATH:condaenvconfigvarssetTRACKER_PATH=$PROJECT_ROOT/db.jsonUsageEnvironment ConfigurationUsetc configto display all the environment variables currently described in your~/.bashrcor Conda environment. To identify which environment variables must be configured for the connector to operate properly, runtc -h.File Operations from Local to Google DriveList Files and FoldersUsetc drive -lsto list all files and folders in your Google Drive Shared directory.Usetc drive -ls -t personalto list all files and folders in your Google Drive "Personal" directory.Open Directoriestc drive -o -p aouexploreopens the "aouexplore" shared drive in your Google Drive.tc drive -o -p aouexplore -s sample_qcopens the "sample_qc" folder in the "aouexplore" shared drive.Upload Files and Folderstc --debug drive --dir up --subdir sample_qcuploads the "sample_qc" folder to the parent directory of your Google Drive root directory, while enabling debug mode.tc drive --dir up --subdir sample_qcperforms the same upload operation without debug mode.File Operations from Local to Google CloudEnvironment SetupYou need to set theCLOUD_ROOTvariable both within your Makefile and Conda environment.List and Download Filestc gcp -lslists all the files and folders in the Google Cloud Storage bucket specified inCLOUD_ROOT.tc -n gcp --dir down --subdir phenotypesdownloads the "phenotypes" folder from your Google Cloud Storage bucket to your local machine.File Operations from Remote Server to Localtc remote -r /gpfs/commons/groups/[Your_Group_Name]/projects/[Your_Project_Name]/ --dir down --subdir preprocessingdownloads the "preprocessing" folder from the specified remote server directory to your local machine.Replace the placeholder values with your specific information where needed.CiteMaintainerTarjinder Singh @ ts3475@cumc.columbia.eduAcknowledgementsRelease Notes