Package: calabar
CalabarUseful tools to make Colab more handy.ContentsInstallationUsageWorking with Google DriveWorking with ArchivesSending Notifications over GmailMiscComplete Usage ExampleInstallation!pip install calabarUsageWorking with Google DriveMount your drive:fromcalabarimportMyDrivedrive=MyDrive('/drive')drive.mount()Copy file/folder from mounted Drive, for example, some data, model checkpoint from the previous training, etc:file2copy='/drive/My Drive/data.tar'dest='/content'drive.cp(file2copy,dest)NOTE:During mounting internal colab tool will create 'My Drive' folder inside/drive,
so don't forget to add 'My Drive' to the file path.Export file from Colab to Drive, for example, training logs, model checkpoints, etc:fromcalabarimportSaveToDriveexport=SaveToDrive()export.to_drive('file_on_colab')export.to_drive('another_file')NOTE:Simply copying to mounted Drive would lead to nothing.Working with ArchivesCurrentlytar/tar.gzandziparchives extracting are supported.fromcalabarimportuntaruntar('data.tar','to_destination')fromcalabarimportunzipunzip('data.zip','to_destination')Sending Notifications over GmailNOTE:To use this option, you need to allow “less secure apps” on your Gmail account:https://www.google.com/settings/security/lesssecureappsSECURITY WARNING:Continuous sending of emails (for example, sending email after each epoch) requires storing your
Gmail account password in memory, allowing you not to relogin to SMTP server each time, when you want to send an email.
BUT, Calabar uses secure TLS connection. Anyway, if you feel unsafe because of "less secure appps" and using
SMTP server, or you just don't like it, you may not use it or use another "less secured" account
(with "less secure apps” option on) and send emails to your primary account, as shown below:fromcalabarimportEmailfrom_addr='johndoe@gmail.com'# email with "allow less secure", this account's SMTP server will be used as a senderto_addrs=['another@gmail.com',]# a list of receiversemail=Email(from_addr,to_addrs)msg_subject='Subject of the email'msg_body='Email body.'email.send(msg_subject,msg_body)usage|contentsMiscfromcalabar.utilsimport*# Print system info, gpu, and installed pytorch versionprint_sysinfo()# Get current GPU RAM usage:print(get_gpu_usage())# Get disk usage:print(get_disk_usage())# Get current time as formatted string in %Y-%m-%d-%H-%M:print(current_time())# Get distro description:print(get_distro_descr())misc|usage|contentsComplete Usage ExampleThe following example shows up how to integrate Calabar with your training pipeline.fromcalabarimportMyDrive,SaveToDrive,untar,unzip,Emailfromcalabar.utilsimport*# Authorize to Drive to allow mountingdrive=MyDrive('/drive')drive.mount()# Authorize to Drive to to allow exporting | NOTE: you can use here another Driveexport=SaveToDrive()from_addr='johndoe@gmail.com'to_addrs=['another@gmail.com']email=Email(from_addr,to_addrs)# Prepare your data# Copy your data:drive.cp('/drive/My Drive/data.tar','/content')# Untar/unzip it:untar('/content/data.tar','/content')# ...# Model definition, training, validation functions etc...# ...# Main loop:# Notification about beginning of the model training processstart_sub='Starting training...'start_msg=f'Training (your model descr) has started at{current_time()}\\nCurrent GPU usage:{get_gpu_usage()}\\nCurrent disk usage:{get_disk_usage()}'email.send(start_sub,start_msg)forepochinn_epochs:train_loss,train_acc=train(...)val_loss,val_acc=validate(...)# Here you're saving your model checkpointcheckpoint_name=f'{epoch}-{val_loss:.3f}-{acc:.3f}-mymodel.checkpoint'# this is just examplesave_checkpoint(f'/content/checkpoints/{checkpoint_name}')# Here you're loading it to your Drive:export.to_drive(f'/content/checkpoints/{checkpoint_name}')# Epoch is over, model checkpoint is exported to your drive, let's notify you about current success:epoch_sub=f'{epoch}/{n_epoch}has finished!'epoch_msg=f'{epoch}/{n_epoch}has finished successfully at{current_time()}\\ntrain_loss:{train_loss:.3f}|val_loss:{val_loss:.3f}|train_acc:{train_acc:.3f}|val_acc:{val_acc:.3f}\\nModel checkpoint{checkpoint_name}successfully loaded to Drive\\nCurrent GPU usage:{get_gpu_usage()}\\nCurrent disk usage:{get_disk_usage()}'email.send(epoch_sub,epoch_msg)# Notification email about model training completion.end_sub='Training has finished!'end_msg=f'Training (your model descr) has finished at{current_time()}\\nCurrent GPU usage:{get_gpu_usage()}\\nCurrent disk usage:{get_disk_usage()}'email.send(end_sub,end_msg)complete usage example|misc|usage|contentsTODOs:tar/zip archivingMore docsMore examples