Package: workspace-puller
Workspace PullerA cli tool for building your python workspace and upload results to the Google Drive or Kaggle.comDescriptionThis tools allows to:download workspace files or archives and extract(bztar, gztar, tar, xztar, zip) it if needed.clone git reposexecute a python filesupload result to the Google Driveupload result to the Kaggle.comInstallation:pip install workspace-pullerUsageworkspace-puller --config_url {url to the config file} --bot_token {telegram bot token}config_urlparameter specifies the URL to the configuration file (Required). Enclose the value in quotes (Recomended).bot_tokenparameter sets the Telegram bot token (Optional).See: Creating a new botAuthorizationTheworkspace-pullerhas two Google Drive authorization methods:local CLI. Just addGoogle Drive folders IDsin config. .remote via telegram chat bot. You need to addGoogle Drive folders IDsin config,telegram chat IDsand specify CLI param--bot_token.
The Telegram bot will send a authorization link, the user must visit link and send the Google token in the response message.Kaggle credentials should be set inside configuration file in theKagglesettings section.After successful authorization Workspace Puller saves credentials inside workspace.ConfiguartionConfiguration file has a YAML format. The file must be accessible through HTTP.Dataset listList of files to download.
Bztar, gztar, tar, xztar, zip files are unzipped by default. If it is not necessary, specify the nodeextract_archives:with a valuefalse. (Default value: true)Example:dataset_list:
  - https://tlk.s3.yandex.net/dataset/TlkAgg2.zip
  - https://tlk.s3.yandex.net/dataset/TlkPersonaChatRus.zip
  - https://tlk.s3.yandex.net/dataset/TlkAgg5.zip
  - https://tlk.s3.yandex.net/dataset/LRWC.zip
  - https://tlk.s3.yandex.net/dataset/WordSenseRus.zip
extract_archives: falseGit reposRepos list for cloning. The repository will clone into a folder with the name specified as the record key.
To add a repository, add a record key with a child nodeurl: {repo url}.If you need a specific branch, specify its name in the child nodebranch:(Optional).Example:repos:
  downloader: 
    url: https://github.com/bantonj/downloader
  pyreadstat: 
    url: https://github.com/Roche/pyreadstat
    branch: devExecuting python codeTo run the code, specify a list of files in the root nodescript_file:Example:script_file:
  - example_file.pyOutput foldersList of folders to upload.Example:output_folder:
  - ./results
  - ./exampleGoogle drive uploadingList of google drive folder IDs to load results into.Example:google_drive:
  - 1MOUoYor2N2TEbJFDKg-4HDKqF-a9AKRq
  - 1MmhIYor2N2CVRJFDKg-4QACKF-a9AUGq
  - 1ATAGart2N2TEbJFDKg-4QfACN-a9NRGqKaggle uploadingKaggle new dataset settingsusername:Kaggle user name (Required)key:Kaggel api token (Required). See:https://github.com/Kaggle/kaggle-api#api-credentialsslug:The URL slug of the dataset you want to create. (Default value:unknown_slug)license:You can specify the following licenses for your dataset (Optional. Default value:CC0-1.0):CC0-1.0:CC0 Public DomainCC-BY-SA-3.0:CC BY-SA 3.0CC-BY-SA-4.0:CC BY-SA 4.0CC-BY-NC-SA-4.0:CC BY-NC-SA 4.0GPL-2.0:GPL 2ODbL-1.0:Database: Open Database, Contents: © Original AuthorsDbCL-1.0:Database: Open Database, Contents: Database Contentscopyright-authors:Data files © Original Authorsother:Other (specified in description)unknown:Unknowndataset_title:Dataset title (Default value:Unknown title)isPrivate:Create publicly (Optional. Default value:true)convertToCsv:Convert tabular files to CSV  (Optional. Default value:true)dirmode:What to do with directories: "skip" - ignore; "zip" - compressed upload; "tar" - uncompressed upload (Optional. Default value:skip)Example:kaggle:
  username: user_name
  key: api_key
  slug: slg_value  
  license: CC0-1.0
  dataset_title: dataset_title
  isPrivate: true
  convertToCsv: true
  dirmode: tarTelegram channelsTelegram chats list for remote Google Drive authorization. The tool sends a message to these chats after the end of work.Example:telegram_channels:
  - 750423741
  - 750423742
  - 750423743Config example:dataset_list:
  - https://tlk.s3.yandex.net/dataset/TlkAgg2.zip
  - https://tlk.s3.yandex.net/dataset/TlkPersonaChatRus.zip
  - https://tlk.s3.yandex.net/dataset/TlkAgg5.zip
  - https://tlk.s3.yandex.net/dataset/LRWC.zip
  - https://tlk.s3.yandex.net/dataset/WordSenseRus.zip
extract_archives: false  
repos:
  downloader: 
    url: https://github.com/bantonj/downloader
  pyreadstat: 
    url: https://github.com/Roche/pyreadstat
    branch: dev
script_file:
  - file.py
output_folder:
  - ./results 
google_drive:
  - NTHDIXBMWNCGFIXBM-NTbmTHD-TGFIT
kaggle:
  username: OEKOIXBT
  key: THDIDHTR4209YDHTR4209ydhcDHT
  slug: wp_test_slg  
  license: CC0-1.0
  dataset_title: wsp_test_dataset
  isPrivate: true
  convertToCsv: true
  dirmode: tar
telegram_channels:
  - 750423741