Step by Step Guide to Fine Tune Models E2E Networks documentation Skip to content E2E Cloud Docs E2E Networks TIR Toggle navigation menu Login Sign Up E2E Networks documentation K Docs E2E Networks TIR Myaccount Getting Started with Myaccount SignUp Process for Indian Customers Customer validation Process for Indian Customers SignUp Process for International Customers Customer Validation Process for International Process Customer Validation Process for Contact Persons Domestic Customer Validation Process FAQs International Customer Validation Process FAQs Sign In Process Release Notes Compute Nodes Virtual Compute Nodes Monitoring 1Click Deployment Active Directory GPU GPU Cloud EQS Introduction How to Create EQS Actions Add Queue under tab Actions for queue service Using SDK Appliance Load Balancer Appliance Auto Scaling Introduction Concepts Define Scale Groups FaaS Function as a Service FaaS How to Create Functions Functions Information Network CDN VPC How to reserve an IP address Reserve IP Pool DNS Custom Reverse DNS Firewall Security Group Tag Security Firewall Bitninja Best Practices SSH Key Management Audit Logs Password Policy 2Factor Authentication Others License Management User Management Purchase SSL CertbotPlugin Apply or Redeem Coupon Code Account Statement Cloud Platform Support Introduction Platform Support Policy FAQ E2Es MyAccount FAQs Code of Conduct of Directors and Senior Management CSR Policy Whistle blower policy Jitsi Meet Mod Evasive Multisite using cPanel Softaculous Copying data from Linux to Windows using WinSCP Migrate MySQL Database Between DBaaS Manage MySQL Abuse Abuse Logs Abuse Attacks Spam Mails Phishing Complaint Copyright Complaint Billing Payment Payment Method EMandate Redeem Coupon Customer Details Updation Setup AutoDebit feature TDS Deduction process Declaration us 206AB Minimum Billing Provisioning and Deprovisioning Process View Invoice and Payment Prepaid Billing Set up Infra Credits Balance Alert Whatsapp notification for Payment reminder Restore Service on a Suspended Account for NonPayment Database Database Database PostgreSQL Parameter Group How to create Parameter Group MongoDB mariaDB Storage Storage Block Storage Object Storage CDP Backup Scalable File System Kubernetes Kubernetes Service Kubernetes Kubernetes Marketplace Monitoring Stack Cert Manager Nginx Ingress Controller Troubleshooting Argo CD Declarative GitOps CD for Kubernetes Deploy to Kubernetes using Argo CD and GitOps Jenkins CI Tool Setup Jenkins On Kubernetes Cluster Ansible CD Tool Istio Service Connecting to Dbaas FAQS Container Registery Container Registry Simplifying Pull Operations with UserFriendly Interface API Developers Guide API SDK E2E CLI Tool Introduction Terraform Terraform Getting Started e2enode Resource Node Plans e2eimage Resource AIML TIR AI Release Notes TIR AI Platform Introduction Getting Started Howto Guides Projects Notebooks Committed Notebook GPU H100 Plans Datasets Model Repository Model Endpoints Pipeline Run Scheduled Run Fine Tuning Models Model Playground Samples and Tutorials API Tokens SSH Key Container Team Features Settings Analytics Billing AIML Tutorials Finetune LLMA with Multiple GPUs Deploy Inference for LLMA 2 Deploy Inference for Codellama Deploy Inference for Stable Diffusion v21 Finetune Stable Diffusion model on TIR Deploy Inference for MPT7BCHAT Custom Containers in TIR Fine Tuning Bloom Natural Language Queries to SQL wth Code Llama E2E Networks documentation Welcome to TIR AI Platform Documentation Step by Step Guide to Fine Tune Models Step by Step Guide to Fine Tune Models Introduction Finetuning a model refers to the process of taking a pretrained machine learning model and further training it on a specific task or dataset to adapt it to the nuances of that particular domain The term is commonly used in the context of transfer learning where a model trained on a large and diverse dataset pretraining is adjusted to perform a specific task or work with a specific dataset finetuning What is FineTuning Finetuning refers to the process of modifying a preexisting pretrained model to cater to a new specific task by training it on a smaller dataset related to the new task This approach leverages the existing knowledge gained from the pretraining phase thereby reducing the need for extensive data and resources In the Context of Neural Networks and Deep Learning In the specific context of neural networks and deep learning finetuning is typically executed by adjusting the parameters of a pretrained model This adjustment is made using a smaller taskspecific dataset The pretrained model having already learned a set of features from a large dataset is further trained on the new dataset to adapt these features to the new task How to Create a FineTuning Job To initiate the FineTuning Job process first the user should navigate to the sidebar section and select Foundation Studio Upon selecting Foundation Studio a dropdown menu will appear featuring an option labeled FineTune Models Upon clicking the FineTune Models option the user will be directed to the Manage FineTuning Jobs page After redirect to the Manage FineTuningJobs On this page users can locate and click on the Create FineTuning Job button or Clickhere button for create FineTunemodels After clicking the Create FineTuning Job button the Create FineTuning Job page will open On this page there are several option such as JobName Model and Hugging Face Token If the user already has an integration with Hugging Face token they can select it from the dropdown options If the user does not have any integration setup with Hugging Face they can click on the Create New link and the Create Integration page will open After adding a token user can move to next stage by clicking on next button If user doesnt have a Hugging Face token They would not be able to access certain services In this case I would need to sign up for an account on Hugging Face and obtain an API token to use their services To obtain a Hugging Face token you can follow these steps Go to the Hugging Face website and create an account if you havent already Once you have created an account log in and go to your account settings Click on the Tokens tab Click on the New Access Token button Give your token a name and select the permissions you want to grant to the token Click on the Create button Your new token will be displayed Make sure to copy it and store it in a safe place as you will not be able to see it again after you close the window Note Some model are available for commercial use but requires access granted by their CustodianAdministrator creatormaintainer of this model You can visit the model card on huggingface to initiate the process How to define DatasetPreparation After defining the Job Model configuration the users can move on to next section for Dataset Preparation The Dataset page will open providing several options such as Select Task Dataset Type Choose a Dataset Validation Split Ratio and Prompt Configuration Once these options are filled the dataset preparation configuration will be set and the user can move to next section Dataset Type In the Dataset Type you can select either CUSTOM or HUGGING FACE as the dataset type The CUSTOM Dataset Type allows training models with userprovided data offering flexibility for unique tasks Alternatively the HuggingFace option provides a variety of preexisting datasets enabling convenient selection and utilization for model training Choose a Dataset CUSTOM If you select dataset type as CUSTOM you have to choose a userdefined dataset by clicking CHOOSE button After clicking on CHOOSE button you will see the below screen if you have already objects in that particular selected dataset To ensure dataset compatibility it is recommended to maintain your data in the jsonl file format This lineoriented JSON format enhances readability and facilitates seamless data processing making it a professional choice for machine learning tasks Please verify and convert your dataset to the jsonl extension prior to model training Note It is crucial to pass the appropriate labels during prompt configuration or you can create a new dataset by clicking click here link Note The listed datasets here use in EOS Bucket for data storage If you click on the click here link you can create a new dataset After clicking youll be able to create a new dataset and click on the CREATE button For Stable Diffusion Model the dataset provided should be of below format One metadata file mapping images with the context text and the list of images in png format foldernamemetadatajsonl foldername0001png foldername0002png foldername0003png structure of metadatajsonl file filename 0001png text This is a first value of a text feature you added to your image filename 0002png text This is a second value of a text feature you added to your image filename 0003png text This is a third value of a text feature you added to your image Note Uploading incorrect dataset format will result into finetuning run failure For Text Models like llama and mistral the dataset provided should be of below formatOne metadata file mapping images with the context text and the list of images in png format input What color is the sky output The sky is blue input Where is the best place to get cloud GPUs output E2E Networks Note Uploading incorrect dataset format will result into finetuning run failure Note Here eg labels input and output will be provided in prompt configuration eg Below is an instruction that describes a task Write a response that appropriately completes the request Instructioninput Responseoutput UPLOAD DATASET After selecting dataset You can upload objects in a particular dataset by selecting dataset and clicking on UPLOAD DATASET button Click on UPLOAD DATASET button and upload objects and click on UPLOAD button Click on OK button After uploading objects to a specific dataset choose a particular file to continue and then click on SUBMIT button HUGGING FACE When opting for the predefined dataset type HUGGING FACE users can conveniently select a dataset from the available collection Subsequently the model training process can be initiated using the chosen dataset streamlining the workflow and enhancing efficiency How to define a Hyperparameter Configuration Upon providing the dataset preparation details users are directed to the Hyperparameter Configuration page This interface allows users to customize the training process by specifying desired hyperparameters thereby facilitating effective hyperparameter tuning The form provided enables the selection of various hyperparameters including but not limited to training type epoch learning rate and max steps Please fill out the form meticulously to optimize the model training process In addition to the standard hyperparameters the configuration page offers advanced options such as batch size and gradient accumulation steps These settings can be utilized to further refine the training process Users are encouraged to explore and employ these advanced options as needed to achieve optimal model performance Upon specifying the advanced settings users are advised to leverage the WandB Integration feature for comprehensive job tracking This involves proceeding to fill in the necessary details in the provided interface By doing so users can effectively monitor and manage the model training process ensuring transparency and control throughout the lifecycle of the job Also they can describe the debug option as desired Once the debug option has been thoroughly addressed users are required to select their preferred machine configuration for the finetuning job Subsequently clicking on the LAUNCH button will initiate or schedule the job depending on the chosen settings To ensure fast and precise training a variety of highperformance GPUs such as Nvidia H100 and A100 are available for selection This allows users to optimize their resources and accelerate the model training process Viewing your Job parameters and Finetuned models On completion of job a FineTuned model will be created and will be shown in models section in lower section of the page This finetuned model repo will contain all checkspoints of model training as well as adapters built during training Users if they desire can also directly go to model repo page under inference to view it If desired the user can view job parameter details in overview section of the job as shown below On this page Introduction What is FineTuning How to Create a FineTuning Job How to define DatasetPreparation Choose a Dataset How to define a Hyperparameter Configuration Viewing your Job parameters and Finetuned models Products TIR AI Platform GPU Dedicated Compute CPU Intensive Cloud High Memory Cloud Linux Smart Dedicated cPanel Linux Cloud Windows Cloud Windows SQL Cloud Plesk Windows Cloud GPU Smart Dedicated Load Balancer Company About Us Meet the Team Become a Partner E2E in Media Testimonials Investors Careers Contact Us Contact Sales Escalation Matrix Service Level Agreement Terms of Service Privacy Policy Refund Policy Policy FAQ Resources Blog Events Service Health Status Help White Papers Ecosystem Enablers Customers Certifications Countries Served FAQs E2E Networks Limited E2E Networks Limited is a NSE Listed AIFirst Hyperscale Cloud Computing Platform CIN Number L72900DL2009PLC341980 Copyright 2023 E2E Networks Limited


Finetune Google Flan UL2 with Multiple GPUs E2E Networks documentation Skip to content E2E Cloud Docs E2E Networks TIR Toggle navigation menu Login Sign Up E2E Networks documentation K Docs E2E Networks TIR Myaccount Getting Started with Myaccount SignUp Process for Indian Customers Customer validation Process for Indian Customers SignUp Process for International Customers Customer Validation Process for International Process Customer Validation Process for Contact Persons Domestic Customer Validation Process FAQs International Customer Validation Process FAQs Sign In Process Release Notes Compute Nodes Virtual Compute Nodes Monitoring 1Click Deployment Active Directory GPU GPU Cloud EQS Introduction Appliance Load Balancer Appliance Auto Scaling Introduction Concepts Define Scale Groups FaaS Function as a Service FaaS Network CDN VPC How to reserve an IP address Reserve IP Pool DNS Custom Reverse DNS Firewall Security Group Tag Security Firewall Bitninja Best Practices SSH Key Management Audit Logs Password Policy 2Factor Authentication Others License Management User Management Purchase SSL CertbotPlugin Apply or Redeem Coupon Code Account Statement Cloud Platform Support Introduction Platform Support Policy FAQ E2Es MyAccount FAQs Code of Conduct of Directors and Senior Management CSR Policy Whistle blower policy Jitsi Meet Mod Evasive Multisite using cPanel Softaculous Copying data from Linux to Windows using WinSCP Migrate MySQL Database Between DBaaS Manage MySQL Abuse Abuse Logs Abuse Attacks Spam Mails Phishing Complaint Copyright Complaint Billing Payment Payment Method EMandate Redeem Coupon Customer Details Updation Setup AutoDebit feature TDS Deduction process Declaration us 206AB Minimum Billing Provisioning and Deprovisioning Process View Invoice and Payment Prepaid Billing Set up Infra Credits Balance Alert Whatsapp notification for Payment reminder Restore Service on a Suspended Account for NonPayment Database Database Database PostgreSQL Parameter Group MongoDB mariaDB Storage Storage Block Storage Object Storage CDP Backup Scalable File System Kubernetes Kubernetes Service Kubernetes Kubernetes Marketplace Monitoring Stack Cert Manager Nginx Ingress Controller Troubleshooting Argo CD Declarative GitOps CD for Kubernetes Deploy to Kubernetes using Argo CD and GitOps Jenkins CI Tool Setup Jenkins On Kubernetes Cluster Ansible CD Tool Istio Service Connecting to Dbaas FAQS Container Registery Container Registery API Developers Guide API SDK E2E CLI Tool Introduction Terraform Terraform Getting Started e2enode Resource Node Plans e2eimage Resource AIML TIR AI Release Notes TIR AI Platform Introduction Getting Started Howto Guides Projects Notebooks Committed Notebook GPU H100 Plans Datasets Model Repository Model Endpoints Pipeline Run Scheduled Run Fine Tuning Models Model Playground Samples and Tutorials API Tokens SSH Key Container Team Features Settings Analytics Billing AIML Tutorials Finetune LLMA with Multiple GPUs Deploy Inference for LLMA 2 Deploy Inference for Codellama Deploy Inference for Stable Diffusion v21 Finetune Stable Diffusion model on TIR Deploy Inference for MPT7BCHAT Custom Containers in TIR Fine Tuning Bloom Natural Language Queries to SQL wth Code Llama E2E Networks documentation Welcome to TIR AI Platform Documentation Tutorials Finetune Google Flan UL2 with Multiple GPUs Finetune Google Flan UL2 with Multiple GPUs In this tutorial we will go through finetune FlanUL2 combining Peft LoRA and Deepspeed on multiple GPUs single machine With A100 80GB cards we can expect the training to finish in 24 hours for 3 epochs FlanUL2 is an encoderdecoder model based on the T5 architecture It is a 20B parameter model finetuned using the Flan prompt tuning and dataset collection The model is was initialized using UL2 checkpoints For more information on UL2 please take a look at the original paper While focus of this article is to cover use of multiple GPUs on TIR platform you may run through the same with a single GPU at the cost of training speed We can also try to fit the model across GPUs using Deepspeeds pipeline parallelism but it does require playing with device map mainly because T5 blocks in UL2 have residual connections which can not be split across GPUs There are workarounds but for the scope of this tutorials we will use Deepspeeds Zero Offloading feature which will allow us to finetune one model copy per GPU through model parallelism Steps 1 Start a GPU Notebook with 4xA10080 Plan and atleast 100GB disk from TIR Dashbaord 2 Install Requirements pip install accelerate transformers peft deepspeed 3 Initiate setup of Accelerate Config accelerate config configfile launcherconfigyaml 4 Choose following parameters In which environment are you running This machine Which type of machine are you using MultiGPU How many different machines will you use 1 Do you wish to optimise your script with torch dynamo No Do you want to use Deepspeed Yes Do you want to specify json for deepspeed config No What should be your Deepspeeds Zero Optimization stage 3 Where to offload CPU optimization stages cpu where to offload parameters cpu How many gradient accumulation steps your are passing to the script 8 Do you want to use gradient clipping no Do you want to save 16bit model no Do you want to enable deepspeedzeroinit no How many gpus should be used for training 4 Do you wish to use FP16 or BF16 no 5 Prepare dataset The finetuning script can work with any csv file here we will use alpaca dataset Create a new file in jypter labs with name preparealpacacsvpy Copy the following contents to this file import json import pandas as pd with openalpacadatajson as f data jsonloadf newformat for i point in enumeratedata no input if lenpointinput 0 inputt Below is an instruction that describes a taskn inputt Write a response that appropriately completes the requestnn inputt f Instructionnpointinstructionnn Response else inputt Below is an instruction that describes a taskn inputt Write a response that appropriately completes the requestnn inputt f Instructionnpointinstructionnn Inputnpointinputnn Response item input inputt output strpointoutput newformatappenditem df pdDataFramenewformat df dfdropna dftocsvalpacadatacsv 6 Open terminal in Jypyter labs Run the following commands to download alpaca set and prepare a csv from it download the json alpaca dataset wget httpsrawgithubusercontentcomtatsulabstanfordalpacamainalpacadatajson run in the terminal shell python preparealpacacsvpy 7 Prepare finetuning script Create a file named trainpy in jupyter labs from file browser and copy the following contents to the file Modified from httpsgithubcomhuggingfacepeftblobmainexamplesconditionalgenerationpeftloraseq2seqacceleratedszero3offloadpy import argparse import gc import logging import os import threading import psutil import torch from accelerate import Accelerator from datasets import loaddataset from deepspeedaccelerator import getaccelerator from peft import LoraConfig TaskType getpeftmodel from torchutilsdata import DataLoader from tqdm import tqdm from transformers import AutoModelForSeq2SeqLM AutoTokenizer getlinearschedulewithwarmup setseed def b2mbx Converting Bytes to Megabytes return intx 220 class TorchTracemalloc Context manager is used to track the peak memory usage of the process def enterself gccollect torchcudaemptycache torchcudaresetmaxmemoryallocated reset the peak gauge to zero selfbegin torchcudamemoryallocated selfprocess psutilProcess selfcpubegin selfcpumemused selfpeakmonitoring True peakmonitorthread threadingThreadtargetselfpeakmonitorfunc peakmonitorthreaddaemon True peakmonitorthreadstart return self def cpumemusedself get resident set size memory for the current process return selfprocessmemoryinforss def peakmonitorfuncself selfcpupeak 1 while True selfcpupeak maxselfcpumemused selfcpupeak if not selfpeakmonitoring break def exitself exc selfpeakmonitoring False gccollect torchcudaemptycache selfend torchcudamemoryallocated selfpeak torchcudamaxmemoryallocated selfused b2mbselfend selfbegin selfpeaked b2mbselfpeak selfbegin selfcpuend selfcpumemused selfcpuused b2mbselfcpuend selfcpubegin selfcpupeaked b2mbselfcpupeak selfcpubegin Handle argument parsing def parseargs parser argparseArgumentParser parseraddargumentmodelpath typestr helpModel path Supports T5UL2 models parseraddargumentdatafilepath typestr defaultsamplecsv helpPath to the already processed dataset parseraddargumentnumepochs typeint default1 helpNumber of epochs to train for parseraddargumentperdevicebatchsize typeint default2 helpBatch size to use for training parseraddargumentinputmaxlength typeint default128 helpMaximum input length to use for generation parseraddargumenttargetmaxlength typeint default128 helpMaximum target length to use for generation parseraddargumentlr typefloat default3e4 helpLearning rate to use for training parseraddargumentseed typeint default42 helpSeed to use for training parseraddargumentinputcolumn typestr defaultinput helpcsv input text column parseraddargumenttargetcolumn typestr defaultoutput helpcsv target text column parseraddargumentsavepath typestr defaultpeftckpt helpSave path args parserparseknownargs return args Main function def main args parseargs textcolumn argsinputcolumn labelcolumn argstargetcolumn lr argslr numepochs argsnumepochs batchsize argsperdevicebatchsize seed argsseed modelnameorpath argsmodelpath datafile argsdatafilepath savepath argssavepath targetmaxlength argstargetmaxlength sourcemaxlength argsinputmaxlength Create dir if it doesnt exist osmakedirssavepath existokTrue Setup logging loggingbasicConfig levelloggingINFO formatasctimes levelnames messages handlers loggingFileHandlerospathjoinsavepath trainingloglog loggingStreamHandler logginginfofArgsn args launch configs accelerator Accelerator peftconfig LoraConfig tasktypeTaskTypeSEQ2SEQLM inferencemodeFalse r8 loraalpha32 loradropout01 setseedseed Save logs only on the main process acceleratoronmainprocess def loginfologging s logginginfos load dataset dataset loaddatasetcsv datafilestrain datafile loginfologging fDataset length lendatasettrain load model model AutoModelForSeq2SeqLMfrompretrainedmodelnameorpath load peft model model getpeftmodelmodel peftconfig modelprinttrainableparameters load tokenizer tokenizer AutoTokenizerfrompretrainedmodelnameorpath def preprocessfunctionsample paddingmaxlength created prompted input inputs sampletextcolumn tokenize inputs modelinputs tokenizer inputs maxlengthsourcemaxlength paddingpadding truncationTrue Tokenize targets with the texttarget keyword argument labels tokenizertexttargetsamplelabelcolumn maxlengthtargetmaxlength paddingpadding truncationTrue If we are padding here replace all tokenizerpadtokenid in the labels by 100 when we want to ignore padding in the loss if padding maxlength labelsinputids l if l tokenizerpadtokenid else 100 for l in label for label in labelsinputids modelinputslabels labelsinputids return modelinputs Prepare and preprocess the dataset with acceleratormainprocessfirst preventing string conversion errors def strconvertexample examplelabelcolumn strexamplelabelcolumn return example datasettrain datasettrainmapstrconvert processeddatasets datasetmap preprocessfunction batchedTrue numproc1 removecolumnsdatasettraincolumnnames loadfromcachefileTrue descRunning tokenizer on dataset acceleratorwaitforeveryone traindataset processeddatasetstrain def collatefnexamples return tokenizerpadexamples paddinglongest returntensorspt traindataloader DataLoader traindataset shuffleTrue collatefncollatefn batchsizebatchsize pinmemoryTrue optimizer optimizer torchoptimAdamWmodelparameters lrlr lr scheduler lrscheduler getlinearschedulewithwarmup optimizeroptimizer numwarmupsteps0 numtrainingstepslentraindataloader numepochs accelerator prepapre model traindataloader optimizer lrscheduler acceleratorprepare model traindataloader optimizer lrscheduler Train the model for epoch in rangenumepochs with TorchTracemalloc as tracemalloc modeltrain totalloss 0 for step batch in enumeratetqdmtraindataloader using accelerator accumulate to perform gradient accumulation with acceleratoraccumulatemodel outputs modelbatch loss outputsloss totalloss lossdetachfloat acceleratorbackwardloss optimizerstep lrschedulerstep optimizerzerograd getacceleratoremptycache Printing the GPU memory usage details loginfologging GPU Peak Memory consumed during train formattracemallocpeaked loginfologging GPU Total Peak Memory consumed during the train format tracemallocpeaked b2mbtracemallocbegin loginfo logging CPU Peak Memory consumed during the train maxbegin formattracemalloccpupeaked loginfologging CPU Total Peak Memory consumed during the train max format tracemalloccpupeaked b2mbtracemalloccpubegin trainepochloss totalloss lentraindataloader trainppl torchexptrainepochloss loginfo logging TRAINING DETAILS loginfologging fepoch trainppl trainepochloss save intermediate checkpoint loginfologging Saving intermediate ckpt acceleratorwaitforeveryone success modelsavecheckpointfsavepath fepoch save peft config peftconfigsavepretrainedospathjoinfsavepath fepoch statusmsg fcheckpointing checkpointfoldersavepath if success loginfologging fSuccess statusmsg else loginfologging fFailure statusmsg loginfologging Training complete if name main main 8 Run the training with accelerate accelerate launch configfile launcherconfigyaml trainpy modelpath googleflanul2 datafilepath alpacadatacsv savepath checkpoint numepochs 1 lr 1e4 perdevicebatchsize 2 inputmaxlength 256 targetmaxlength 256 9 Prepare checkpoint for inference Deepspeed will create sharded model files and a zerotofp32py in the checkpoint folder Run the following command to convert the checkpoints to Pytorch bin file replace 0 with the latest checkpoints number python checkpointzerotofp32py checkpoint checkpoint0adaptermodelbin 10 Load finetuned model for inference Start a new notebook to run this code from a notebook cell from transformers import AutoModelForSeq2SeqLM from peft import PeftModel PeftConfig peftmodelid checkpoint0 basemodelid Googleflanul2 model AutoModelForSeq2SeqLMfrompretrainedbasemodelid The original model path model PeftModelfrompretrainedmodel peftmodelid The finetuned model path You may also further merge the base weights with LoRA checkpoints Conclusion In this tutorial we have finetuned a large language model with multiple GPUs using Deepspeed and LoRA A similar apporach can be followed for finetuning other models On this page Steps Conclusion Products TIR AI Platform GPU Dedicated Compute CPU Intensive Cloud High Memory Cloud Linux Smart Dedicated cPanel Linux Cloud Windows Cloud Windows SQL Cloud Plesk Windows Cloud GPU Smart Dedicated Load Balancer Company About Us Meet the Team Become a Partner E2E in Media Testimonials Investors Careers Contact Us Contact Sales Escalation Matrix Service Level Agreement Terms of Service Privacy Policy Refund Policy Policy FAQ Resources Blog Events Service Health Status Help White Papers Ecosystem Enablers Customers Certifications Countries Served FAQs E2E Networks Limited E2E Networks Limited is a NSE Listed AIFirst Hyperscale Cloud Computing Platform CIN Number L72900DL2009PLC341980 Copyright 2023 E2E Networks Limited


How to Launch GPU H100 notebook E2E Networks documentation Skip to content E2E Cloud Docs E2E Networks TIR Toggle navigation menu Login Sign Up E2E Networks documentation K Docs E2E Networks TIR Myaccount Getting Started with Myaccount SignUp Process for Indian Customers Customer validation Process for Indian Customers SignUp Process for International Customers Customer Validation Process for International Process Customer Validation Process for Contact Persons Domestic Customer Validation Process FAQs International Customer Validation Process FAQs Sign In Process Release Notes Compute Nodes Virtual Compute Nodes Monitoring 1Click Deployment Active Directory GPU GPU Cloud EQS Introduction How to Create EQS Actions Add Queue under tab Actions for queue service Using SDK Appliance Load Balancer Appliance Auto Scaling Introduction Concepts Define Scale Groups FaaS Function as a Service FaaS How to Create Functions Functions Information Network CDN VPC How to reserve an IP address Reserve IP Pool DNS Custom Reverse DNS Firewall Security Group Tag Security Firewall Bitninja Best Practices SSH Key Management Audit Logs Password Policy 2Factor Authentication Others License Management User Management Purchase SSL CertbotPlugin Apply or Redeem Coupon Code Account Statement Cloud Platform Support Introduction Platform Support Policy FAQ E2Es MyAccount FAQs Code of Conduct of Directors and Senior Management CSR Policy Whistle blower policy Jitsi Meet Mod Evasive Multisite using cPanel Softaculous Copying data from Linux to Windows using WinSCP Migrate MySQL Database Between DBaaS Manage MySQL Abuse Abuse Logs Abuse Attacks Spam Mails Phishing Complaint Copyright Complaint Billing Payment Payment Method EMandate Redeem Coupon Customer Details Updation Setup AutoDebit feature TDS Deduction process Declaration us 206AB Minimum Billing Provisioning and Deprovisioning Process View Invoice and Payment Prepaid Billing Set up Infra Credits Balance Alert Whatsapp notification for Payment reminder Restore Service on a Suspended Account for NonPayment Database Database Database PostgreSQL Parameter Group How to create Parameter Group MongoDB mariaDB Storage Storage Block Storage Object Storage CDP Backup Scalable File System Kubernetes Kubernetes Service Kubernetes Kubernetes Marketplace Monitoring Stack Cert Manager Nginx Ingress Controller Troubleshooting Argo CD Declarative GitOps CD for Kubernetes Deploy to Kubernetes using Argo CD and GitOps Jenkins CI Tool Setup Jenkins On Kubernetes Cluster Ansible CD Tool Istio Service Connecting to Dbaas FAQS Container Registery Container Registry Simplifying Pull Operations with UserFriendly Interface API Developers Guide API SDK E2E CLI Tool Introduction Terraform Terraform Getting Started e2enode Resource Node Plans e2eimage Resource AIML TIR AI Release Notes TIR AI Platform Introduction Getting Started Howto Guides Projects Notebooks Committed Notebook GPU H100 Plans Datasets Model Repository Model Endpoints Pipeline Run Scheduled Run Fine Tuning Models Model Playground Samples and Tutorials API Tokens SSH Key Container Team Features Settings Analytics Billing AIML Tutorials Finetune LLMA with Multiple GPUs Deploy Inference for LLMA 2 Deploy Inference for Codellama Deploy Inference for Stable Diffusion v21 Finetune Stable Diffusion model on TIR Deploy Inference for MPT7BCHAT Custom Containers in TIR Fine Tuning Bloom Natural Language Queries to SQL wth Code Llama E2E Networks documentation Welcome to TIR AI Platform Documentation How to Launch GPU H100 notebook How to Launch GPU H100 notebook Select from prebuilt or one of your own images Next you choose GPU H100 plan and click on create button Notebook options Disk Size Each TIR Notebook can have a disk size upto 1000GB The default is 30GB The selected disk will be mounted at homejovyan in your notebook environment We recommend using this path as your workspace so in case of restarts your content will be persistent Since TIR is containernative the changes that you make to any other paths on the notebook will not be persisted on restarts You can extend the disk size after the start of notebook as well This workspace will be deleted when the associated notebook is deleted Note Please raise a support ticket if you need more than 1TB of disk workspace Local NVME Storage Only available for GPU H100 plans This fast local storage will be available at mntlocal andonly for the duration of run We recommend using this path when you need faster writes eg save model checkpoints or reads Be sure to move this data to EOS bucket or under homejovyan before shutting down the notebook This type of storage is fixed and can not be expanded at anytime during the notebook cycle Note By default you receive 1000 GB The size of the local storage is fixed and cannot be altered Plan Pricing You can choose between an hourly or committed plan We recommend using committed plans as they offer discounts and also may offer access to local NVME storage for H100 plans only Notebook Image TIR environments are containernative You can use prebuilt images with well known frameworks like pytorch tranformers or customise the prebuilt images You can make your own images TIRcompatible using image builder utility We recommend starting with prebuilt images In case you need to install packages from pip or aptget we recommend doing so from a jupyter notebook ipynb or maintaining requirementstxt Delete Notebook When a notebook is deleted all the resources associated with it will be deleted including the workspace disk Steps to create GPU H100 notebook To create a Notebook you have to click on Create Notebook which is at the left corner of the page After clicking on the Create Notebook button a page will appear now enter Notebook name choose the Notebook Image and select the GPU H100 machine to run your service and then choose the plan After that click on Create button After entering all the details Disk Size Datasets and ssh key you will click on the create button for creating a notebook After clicking on the create button your GPU H100 will be created and it will be shown like this Notebook Details Overview You can see the Notebook Details Plan Details under Overview tab Disk Size You can see the details disk size and also You can change the Disk size as per your requirements For updating the disk size you have to change the disk size and then click on update button Metrics You can see the Metrics graph in CPU Utilization Memory Utilization Interval You can see the one month activity as per your requirement in days hours Associated Datasets You can also see the Associated Datasets with two different datasets Mounted Unmounted You can also Unmount SSH Key You can see the SSH Key Details under ssh key tab Note Only one SSH key can be added to a notebook Here are the Steps to verify the configuration of the GPU H100 notebook using the terminal Create your notebook with ssh key You can access your GPU H100 notebook by clicking SSH Access icon After clicking this ssh access icon you can able to see the command details section To access your GPU H100 notebook use this command ssh rootip To check the GPU resources use this command nvidiasmi Or you can verify the configuration of the GPU H100 notebook using the Lab URL To launch GPU H100 notebook click on jupter After clicking on jupter you can see that page and click on Python3 To check the GPU resources use this command nvidiasmi and press shift enter Notebook Actions You can see the actions like Launch Notebook Stop Update Notebook Delete Launch Notebook After clicking on Launch Notebook Notebook will be launched and it should be visible like this Stop Note You can not stop GPU H100 notebook Delete Notebook For Deleting the Notebook you have to click on Delete button On this page Notebook options Steps to create GPU H100 notebook Notebook Details Overview Disk Size Metrics Associated Datasets SSH Key Here are the Steps to verify the configuration of the GPU H100 notebook using the terminal Or you can verify the configuration of the GPU H100 notebook using the Lab URL Notebook Actions Launch Notebook Stop Delete Notebook Products TIR AI Platform GPU Dedicated Compute CPU Intensive Cloud High Memory Cloud Linux Smart Dedicated cPanel Linux Cloud Windows Cloud Windows SQL Cloud Plesk Windows Cloud GPU Smart Dedicated Load Balancer Company About Us Meet the Team Become a Partner E2E in Media Testimonials Investors Careers Contact Us Contact Sales Escalation Matrix Service Level Agreement Terms of Service Privacy Policy Refund Policy Policy FAQ Resources Blog Events Service Health Status Help White Papers Ecosystem Enablers Customers Certifications Countries Served FAQs E2E Networks Limited E2E Networks Limited is a NSE Listed AIFirst Hyperscale Cloud Computing Platform CIN Number L72900DL2009PLC341980 Copyright 2023 E2E Networks Limited


E2E Networks payment options E2E Networks documentation Skip to content E2E Cloud Docs E2E Networks TIR Toggle navigation menu Login Sign Up E2E Networks documentation K Docs E2E Networks TIR Myaccount Getting Started with Myaccount SignUp Process for Indian Customers Customer validation Process for Indian Customers SignUp Process for International Customers Customer Validation Process for International Process Customer Validation Process for Contact Persons Domestic Customer Validation Process FAQs International Customer Validation Process FAQs Sign In Process Release Notes Compute Nodes Virtual Compute Nodes Monitoring 1Click Deployment Active Directory GPU GPU Cloud EQS Introduction How to Create EQS Actions Add Queue under tab Actions for queue service Using SDK Appliance Load Balancer Appliance Auto Scaling Introduction Concepts Define Scale Groups FaaS Function as a Service FaaS How to Create Functions Functions Information Network CDN VPC How to reserve an IP address Reserve IP Pool DNS Custom Reverse DNS Firewall Security Group Tag Security Firewall Bitninja Best Practices SSH Key Management Audit Logs Password Policy 2Factor Authentication Others License Management User Management Purchase SSL CertbotPlugin Apply or Redeem Coupon Code Account Statement Cloud Platform Support Introduction Platform Support Policy FAQ E2Es MyAccount FAQs Code of Conduct of Directors and Senior Management CSR Policy Whistle blower policy Jitsi Meet Mod Evasive Multisite using cPanel Softaculous Copying data from Linux to Windows using WinSCP Migrate MySQL Database Between DBaaS Manage MySQL Abuse Abuse Logs Abuse Attacks Spam Mails Phishing Complaint Copyright Complaint Billing Payment Payment Method EMandate Redeem Coupon Customer Details Updation Setup AutoDebit feature TDS Deduction process Declaration us 206AB Minimum Billing Provisioning and Deprovisioning Process View Invoice and Payment Prepaid Billing Set up Infra Credits Balance Alert Whatsapp notification for Payment reminder Restore Service on a Suspended Account for NonPayment Database Database Database PostgreSQL Parameter Group How to create Parameter Group MongoDB mariaDB Storage Storage Block Storage Object Storage CDP Backup Scalable File System Kubernetes Kubernetes Service Kubernetes Kubernetes Marketplace Monitoring Stack Cert Manager Nginx Ingress Controller Troubleshooting Argo CD Declarative GitOps CD for Kubernetes Deploy to Kubernetes using Argo CD and GitOps Jenkins CI Tool Setup Jenkins On Kubernetes Cluster Ansible CD Tool Istio Service Connecting to Dbaas FAQS Container Registery Container Registry Simplifying Pull Operations with UserFriendly Interface API Developers Guide API SDK E2E CLI Tool Introduction Terraform Terraform Getting Started e2enode Resource Node Plans e2eimage Resource AIML TIR AI Release Notes TIR AI Platform Introduction Getting Started Howto Guides Projects Notebooks Committed Notebook GPU H100 Plans Datasets Model Repository Model Endpoints Pipeline Run Scheduled Run Fine Tuning Models Model Playground Samples and Tutorials API Tokens SSH Key Container Team Features Settings Analytics Billing AIML Tutorials Finetune LLMA with Multiple GPUs Deploy Inference for LLMA 2 Deploy Inference for Codellama Deploy Inference for Stable Diffusion v21 Finetune Stable Diffusion model on TIR Deploy Inference for MPT7BCHAT Custom Containers in TIR Fine Tuning Bloom Natural Language Queries to SQL wth Code Llama E2E Networks documentation E2E Networks Billing and Payment Information E2E Networks payment options E2E Networks payment options Important Note Our Payment Gateway Partner has recently migrated all Virtual accounts to RBL Bank IFSC Code RATN0VAAPIS Please check your updated virtual account details under the Billing Section of your Myaccount Billing Billing Profile Virtual Bank Account Details and pay accordingly only in RBL Bank Virtual accounts E2E Networks accepts payments through the following methods Option Name Description AutoPay Automatic charging from your credit or debit card based on the standing instructions given by you PayNow Onetime payment through the payment gateway using credit card debit card net banking ewallets and unified payment interfaceUPI Virtual Account Bank Transfer Onetime payment through bank transfer NEFT RTGS IMPS using your virtual account with E2E Networks Ltd This option may not work for bank transfer from outside India AutoPay PLEASE NOTE in accordance with RBI guidelines on the processing of recurring payments existing standing instructions received for Autopay shall not work postSeptember 30 2021 and accordingly we will deactivate these existing instructions for auto pay on September 30 2021 end of the day Our payment gateway partner has been able to make live a few banks cards for the processing of recurring payments so far in accordance with these new guidelines So postOctober 1 2021 Customers may add the cards which have been made live by our payment gateway partner As of now the following cards are live SNo Bank Debit Card Credit Card 1 HSBC NA 2 Equitas Small Finance Bank NA 3 City Union Bank NA 4 OneCard NA 5 Karur Vysya Bank The AutoPay method allows you to set up standing instructions on your creditdebit card to allow E2E Networks Ltd to automatically charge your card This ensures uninterrupted service by eliminating the risk of suspension due to delayed payments This also saves you the hassle of regular tracking and making periodic payments on the MyAccount portal Payment gateway window E2E Networks has integrated with the Razorpay payment gateway and Stripe payment gateway for Autopay only The behaviour for Prepaid Customers For Enabling Autopay firstly you need to click Billing from side nav bar and click on Autopay Smart Top Up SmartTopup is a convenience feature It gives you a piece of mind by ensuring that your account infracredit balance is maintained When you keep SmartTopup enabled with Autopay service The account balance quickly gets refilled by making an attempt to charge your saved payment method as soon as the infracredit balance goes below 2000 As for example if your last 7 days usage 5000 then your account will be auto recharged with the value of 5000 if your infra credit balance goes below 2000 This is better than setting a fixed refill value as sometimes when you launch new Node instances or other services the periodic burn rate of the infracredits also increases The refill amount value needs to be smartly identified based on current usage trend of your account and upcoming charges like minimumbilling charges license renewal committed node renewal etc For enabling Autopay Smart Top Up you have to click on Autopay you need to check on the checkbox of Enable Autopay after clicking enable autopay checkbox then the smart top up will be by default checked and Enabled then you have to click on Save button If user wants to disable the SmartTopup then click on enable smart Top Up checkbox and click on Save button to save the changes After clicking on save button popup will come on screen for selecting payment gateway as of now E2E providing two payment gateway method azorpay payment gateway and Stripe payment gateway for Autopay You have to select whatever payment gateway After selecting payment gateway Payment gateway window will open and you need to enter details of the card and pay 1 Then your card will be saved for Autopay For Adding more card you need to click on Add CreditDebit card button and select the payment gatewayHere we have choose Stripe and enter details and click on save button then your card will be save Note For Indian users the autopay feature initiates a transaction on the users saved card A predebit notification is sent by the bank 24 hours prior to the actual debit When the amount value is greater than INR 15000 the customer would have to give an additional approval to the bank to allow the transaction Note Only Mastercard and Visa cards are eligible for processing through the Stripe gateway Note Your Card detail will be saved by the payment gateway partner Click on Pay 1 and you will be redirected to the bank page to complete a transaction of 1 which will be deducted by the payment gateway to verify the credit card or debit card On completing this transaction The deducted 1 will be refunded within 14 working days to the card after the verification Thats it Your Autopay has been set up using the card that will be used for all your payments If youre using the prepaid billing method then you need to follow additional steps You need to tick the checkbox that says Enable AutoPay to automatically buy Infra credits for MyAccount In the When MyAccount Infra credits balance reaches field you need to enter the threshold value In the then buy field enter the number of infra credits you would like to purchase automatically when the infra credit balance reaches the threshold amount Then Click on Save Note There is a minimum value 2000 which you can enter in the then buy field Whenever your account is topped up automatically you will receive email alerts from your bank If the transaction fails on the primary card provided by you an email notification will be sent to you You can add multiple cards by following the above steps Note You need to have at least one active card added under Manage Cards If there is no card added automatic payment setup cannot be done If the transaction fails on the primary card provided by you an email notification will be sent to you The behaviour for Postpaid Customers For Enabling Autopay firstly you need to click Billing from side nav bar and click on Autopay After that you have to click on Add CreditDebit card button After clicking on Add CreditDebit card button you have to select payment gateway After selecting payment gateway Payment gateway window will open and you need to enter details of the card and pay 1 Then your card will be saved for Autopay Note Your Card detail will be saved by the payment gateway partner Click on Pay 1 and you will be redirected to the bank page to complete a transaction of 1 which will be deducted by the payment gateway to verify the credit card or debit card On completing this transaction The deducted 1 will be refunded within 14 working days to the card after the verification Thats it Your Autopay has been set up using the card that will be used for all your payments PayNow PayNow method is used for making a onetime payment of any amount using a credit card a debit card net banking ewallets or a unified payment interfaceUPI Login to MyAccount at httpsmyaccounte2enetworkscom On the left side of the MyAccount dashboard click on the PayNow submenu available under the Billing section The behavior for Prepaid User For a prepaid user the amount paid shall be used to buy infracredits In the Credits to be purchased field you need to enter the number of Infra credits you want to purchase The Net amount to be paid shows the amount to be paid including GST Click on the Proceed to Pay button to launch the payment gateway window The behavior for postpaid user For postpaid users the screen as shown below shall appear where a customer can make any payment The payment made by a customer is first used to settle any unpaid invoices and the remaining amount is considered as an advance payment in our system The screen automatically populates amounts from the last bill any advance payments are done before and any outstanding amount related to your account The Payment Amount field will be autopopulated based on the below calculation Latest Bill Amount Advance Amount Current Outstanding This calculation excludes the TDS Click here to know more about the TDS deduction User is allowed to change the amount that is autopopulated in the Payment Amount field Click on the Proceed to Pay button to proceed to the payment gateway window Payment gateway window E2E Networks has integrated with the Razorpay payment gateway A payment popup box appears where you can choose from any of the following options like credit and debit cards UPI net banking and wallets etc Card Currently our payment gateway supports Indian and International credit and debit cards from Visa Mastercard and American Express While making payments using a card you can select the Remember Card checkbox to save the card The card details are not saved by E2E Networks Ltd The card details are saved by payment gateway Razorpay The card details can be deleted by logging into the Razorpay website You can access saved cards while you make payments OTP based authentication shall be performed to access saved cards The OTP will be sent to your primary contacts mobile number Netbanking You can pay using the Netbanking interfaces of various banks After selecting your preferred bank from the list of banks available you will be taken to the respective banks login interface to complete the payment Wallet You can pay using ewallets Currently Amazon Pay Freecharge Ola Money and PayZapp are supported After selecting your preferred wallet you will be taken to the respective wallets interface to complete the payment Note While using the PayNow option Razorpay may show some offers from its partners such as cashback or discounts Such offers and benefits are governed by the terms and conditions set by the providers of such offers and in no way associated with E2E Networks Ltd UPI You can enter your UPI address to get a payment request directly in your UPI app You can pay using UPI apps such as BHIM Google Pay WhatsApp Paytm PhonePe and others Alternatively you can also scan the payment QR code and pay using your UPI address EMI Debit EMI is an EMI method using which customers can make payments on EMI using their debit cards Virtual Account Bank Transfer To make payments through IMPS NEFT and RTGS the transfer can be made through a virtual bank account associated with your MyAccount The Virtual Account Bank Transfer feature is available to all E2Es customers prepaid as well as postpaid One of the advantages of a virtual bank account is that the settlement of the transaction will be processed automatically through the payment gateway Kindly note that each of your MyAccount will have unique virtual bank account details and the payment is done using Virtual Account Bank Transfer will be processed only for the respective MyAccount To check the virtual bank account details for your MyAccount follow the steps Login to MyAccount at httpsmyaccounte2enetworkscom On the left side of the MyAccount dashboard and click on the billing profile submenu available under the Billing section Then click on the Virtual Bank Account Details tab Note The virtual bank account detail cannot be used for making payments through Cheque E2E Networks payment options for International customers E2E Networks accepts payments through the following methods PayNow PayNow method is used for making a onetime payment of any amount using a credit card a debit card net banking ewallets or a unified payment interfaceUPI Login to MyAccount at httpsmyaccounte2enetworkscom On the left side of the MyAccount dashboard click on the PayNow submenu available under the Billing section The behavior for Prepaid User For a prepaid user the amount paid shall be used to buy infracredits In the Credits to be purchased field you need to enter the number of Infra credits you want to purchase The Net amount to be paid shows the amount to be paid including GST Click on the Proceed to Pay button to launch the payment gateway window Payment gateway window E2E Networks has integrated with the Stripe payment gateway A payment popup box appears Currently our payment gateway supports International credit and debit cards from Visa Mastercard and American Express Card After filling all the details like Card number Expiration dateCVV noCountry and Zip Code Then click on the Pay now tab then another popup will open and it will show a congratulation message AutoPay The AutoPay method allows you to set up standing instructions on your creditdebit card to allow E2E Networks Ltd to automatically charge your card This ensures uninterrupted service by eliminating the risk of suspension due to delayed payments This also saves you the hassle of regular tracking and making periodic payments on the MyAccount portal Login to MyAccount at httpsmyaccounte2enetworkscom On the left side of the MyAccount dashboard click on the Auto Pay submenu available under the Billing section After clicking on the Auto Pay tab in Manage AutoPay we can enable the following options Enable AutoPay to automatically buy Infra credits for MyAccount Enable low Infra credits notification I also want an SMS notification After clicking on Add CreditDebit Cards Payment gateway window will appear Payment gateway window E2E Networks has integrated with the Stripe payment gateway A payment popup box appears where you can choose from any of the following options like Visa Mastercard and American Express Currently our payment gateway supports International credit and debit cards from Visa Mastercard and American Express After filling all the details likeCard numberExpiration dateCVV noCountry and Zip Code Then click on Pay now tab then another popup will open and it will show a Setting Saved pop up will appear then read the instructions and click onOkayButton Note Your Card detail will be saved by the payment gateway partner Click on Pay 1 and you will be redirected to the bank page to complete a transaction of 1 which will be deducted by the payment gateway to verify the credit card or debit card On completing this transaction The deducted 1 will be refunded within 14 working days to the card after the verification Thats it Your Autopay has been set up using the card that will be used for all your payments Note There is a minimum value 2000 which you can enter in the then buy field Whenever your account is topped up automatically you will receive email alerts from your bank If the transaction fails on the primary card provided by you an email notification will be sent to you You can add multiple cards by following the above steps Note You need to have at least one active card added under Manage Cards If there is no card added automatic payment setup cannot be done If the transaction fails on the primary card provided by you an email notification will be sent to you On this page AutoPay Payment gateway window The behaviour for Prepaid Customers Smart Top Up The behaviour for Postpaid Customers PayNow The behavior for Prepaid User The behavior for postpaid user Payment gateway window Card Netbanking Wallet UPI EMI Virtual Account Bank Transfer E2E Networks payment options for International customers PayNow The behavior for Prepaid User Payment gateway window Card AutoPay Payment gateway window Products TIR AI Platform GPU Dedicated Compute CPU Intensive Cloud High Memory Cloud Linux Smart Dedicated cPanel Linux Cloud Windows Cloud Windows SQL Cloud Plesk Windows Cloud GPU Smart Dedicated Load Balancer Company About Us Meet the Team Become a Partner E2E in Media Testimonials Investors Careers Contact Us Contact Sales Escalation Matrix Service Level Agreement Terms of Service Privacy Policy Refund Policy Policy FAQ Resources Blog Events Service Health Status Help White Papers Ecosystem Enablers Customers Certifications Countries Served FAQs E2E Networks Limited E2E Networks Limited is a NSE Listed AIFirst Hyperscale Cloud Computing Platform CIN Number L72900DL2009PLC341980 Copyright 2023 E2E Networks Limited


SSH Keys Management E2E Networks documentation Skip to content E2E Cloud Docs E2E Networks TIR Toggle navigation menu Login Sign Up E2E Networks documentation K Docs E2E Networks TIR Myaccount Getting Started with Myaccount SignUp Process for Indian Customers Customer validation Process for Indian Customers SignUp Process for International Customers Customer Validation Process for International Process Customer Validation Process for Contact Persons Domestic Customer Validation Process FAQs International Customer Validation Process FAQs Sign In Process Release Notes Compute Nodes Virtual Compute Nodes Monitoring 1Click Deployment Active Directory GPU GPU Cloud EQS Introduction How to Create EQS Actions Add Queue under tab Actions for queue service Using SDK Appliance Load Balancer Appliance Auto Scaling Introduction Concepts Define Scale Groups FaaS Function as a Service FaaS How to Create Functions Functions Information Network CDN VPC How to reserve an IP address Reserve IP Pool DNS Custom Reverse DNS Firewall Security Group Tag Security Firewall Bitninja Best Practices SSH Key Management Audit Logs Password Policy 2Factor Authentication Others License Management User Management Purchase SSL CertbotPlugin Apply or Redeem Coupon Code Account Statement Cloud Platform Support Introduction Platform Support Policy FAQ E2Es MyAccount FAQs Code of Conduct of Directors and Senior Management CSR Policy Whistle blower policy Jitsi Meet Mod Evasive Multisite using cPanel Softaculous Copying data from Linux to Windows using WinSCP Migrate MySQL Database Between DBaaS Manage MySQL Abuse Abuse Logs Abuse Attacks Spam Mails Phishing Complaint Copyright Complaint Billing Payment Payment Method EMandate Redeem Coupon Customer Details Updation Setup AutoDebit feature TDS Deduction process Declaration us 206AB Minimum Billing Provisioning and Deprovisioning Process View Invoice and Payment Prepaid Billing Set up Infra Credits Balance Alert Whatsapp notification for Payment reminder Restore Service on a Suspended Account for NonPayment Database Database Database PostgreSQL Parameter Group How to create Parameter Group MongoDB mariaDB Storage Storage Block Storage Object Storage CDP Backup Scalable File System Kubernetes Kubernetes Service Kubernetes Kubernetes Marketplace Monitoring Stack Cert Manager Nginx Ingress Controller Troubleshooting Argo CD Declarative GitOps CD for Kubernetes Deploy to Kubernetes using Argo CD and GitOps Jenkins CI Tool Setup Jenkins On Kubernetes Cluster Ansible CD Tool Istio Service Connecting to Dbaas FAQS Container Registery Container Registry Simplifying Pull Operations with UserFriendly Interface API Developers Guide API SDK E2E CLI Tool Introduction Terraform Terraform Getting Started e2enode Resource Node Plans e2eimage Resource AIML TIR AI Release Notes TIR AI Platform Introduction Getting Started Howto Guides Projects Notebooks Committed Notebook GPU H100 Plans Datasets Model Repository Model Endpoints Pipeline Run Scheduled Run Fine Tuning Models Model Playground Samples and Tutorials API Tokens SSH Key Container Team Features Settings Analytics Billing AIML Tutorials Finetune LLMA with Multiple GPUs Deploy Inference for LLMA 2 Deploy Inference for Codellama Deploy Inference for Stable Diffusion v21 Finetune Stable Diffusion model on TIR Deploy Inference for MPT7BCHAT Custom Containers in TIR Fine Tuning Bloom Natural Language Queries to SQL wth Code Llama E2E Networks documentation Security in E2E Networks SSH Keys Management SSH Keys Management Introduction Setting up SSHbased security to access your server is a much more effective way than the use of a manual root password Cracking the security system of a node depending on SSH keys is nearly impossible since it secures your node in a more sophisticated way by the use of encoded keys Why is passwordbased authentication vulnerable A server can authenticate grant access to the users with different access methods The most basic of these is a passwordbased authentication which is easy to use but isnt the most secure Modern processing power combined with automated scripts makes bruteforcing a passwordprotected account very possible since passwords generally are not complex SSH keys prove to be a reliable and secure alternative What are SSH keys SSH key pairs are two cryptographically secure keys that can be used to authenticate a client to an SSH server Each key pair consists of a public key and a private key The private key is retained by the client on his local machine and should be kept secret Any compromise of the private key will allow the attacker to log into servers that are configured with the associated public key without additional authentication As an additional precaution the key can be encrypted on disk with a passphrase The public key is uploaded onto the remote server that you want to be able to log into with SSH When a client attempts to authenticate using SSH keys the server can test the client on whether they are in possession of the private key If the keypair matches then a shell session is spawned or the requested command is executed How do SSH keys work A key pair will be generated on your local PC Generating a key pair provides you with two long string of characters a public and a private key The public key will be added to your node The corresponding private key pair will be saved on your local PC Every time you access your node the SSH system will lookup for the private key pair of the public key added to it The system will unlock only when the two keys match You can also disable the root password after the SSH keys are set up Note Secure the private key Make sure that you add the public key to the servers and the private key is saved in a secure location on your PC Manage SSH Keys This guide shows you how to access manage your SSH keys into E2E Networks My Account portal Logging into E2E Networks My Account Please go to My Account and log in using your credentials set up at the time of creating and activating the E2E Networks My Account Navigate to SSH Keys After you log in to the E2E Networks My Account you can click on the left side of the MyAccount dashboard click on the SSH Keys submenu available under the services menu AddDelete SSH Keys If you do not have SSH keys and want to create an SSH key pair to access your node then follow the tutorial for your computer OS MAC Windows Linux ADD SSH Keys Click on Add Key button You need to label your SSH key optional field for easy identification purpose You can either load the file to add public SSH Key by clicking the Load from file or paste the contents of your public SSH key copy it as it is and paste it in in the SSH Key content field Note Before you paste your public SSH keys into your content field you must check the format of each public SSH key file that you plan to add After adding the public key you need to click the Add Key button you will be automatically routed to the Manage SSH Keys page Similarly you can store multiple SSH Public Keys into My Account may assign them to the E2E node which will be accessible to multiple users with different SSH keypairs Delete SSH Keys To remove a public key click on the delete button which will permanently delete your public key Note Any user accessing the node from the SSH keypair will not be able to access the node once the key is removed from My Account Using an SSH Key Now the SSH key is added to your My Account which can be used with any new virtual compute node that you create in the future by simply selecting the public key during a virtual compute node creation process You can also refer enabledisable passwordbased authentication for SSH access to the server On this page Introduction Manage SSH Keys Logging into E2E Networks My Account Navigate to SSH Keys AddDelete SSH Keys ADD SSH Keys Delete SSH Keys Using an SSH Key Products TIR AI Platform GPU Dedicated Compute CPU Intensive Cloud High Memory Cloud Linux Smart Dedicated cPanel Linux Cloud Windows Cloud Windows SQL Cloud Plesk Windows Cloud GPU Smart Dedicated Load Balancer Company About Us Meet the Team Become a Partner E2E in Media Testimonials Investors Careers Contact Us Contact Sales Escalation Matrix Service Level Agreement Terms of Service Privacy Policy Refund Policy Policy FAQ Resources Blog Events Service Health Status Help White Papers Ecosystem Enablers Customers Certifications Countries Served FAQs E2E Networks Limited E2E Networks Limited is a NSE Listed AIFirst Hyperscale Cloud Computing Platform CIN Number L72900DL2009PLC341980 Copyright 2023 E2E Networks Limited


Containers E2E Networks documentation Skip to content E2E Cloud Docs E2E Networks TIR Toggle navigation menu Login Sign Up E2E Networks documentation K Docs E2E Networks TIR Myaccount Getting Started with Myaccount SignUp Process for Indian Customers Customer validation Process for Indian Customers SignUp Process for International Customers Customer Validation Process for International Process Customer Validation Process for Contact Persons Domestic Customer Validation Process FAQs International Customer Validation Process FAQs Sign In Process Release Notes Compute Nodes Virtual Compute Nodes Monitoring 1Click Deployment Active Directory GPU GPU Cloud EQS Introduction How to Create EQS Actions Add Queue under tab Actions for queue service Using SDK Appliance Load Balancer Appliance Auto Scaling Introduction Concepts Define Scale Groups FaaS Function as a Service FaaS How to Create Functions Functions Information Network CDN VPC How to reserve an IP address Reserve IP Pool DNS Custom Reverse DNS Firewall Security Group Tag Security Firewall Bitninja Best Practices SSH Key Management Audit Logs Password Policy 2Factor Authentication Others License Management User Management Purchase SSL CertbotPlugin Apply or Redeem Coupon Code Account Statement Cloud Platform Support Introduction Platform Support Policy FAQ E2Es MyAccount FAQs Code of Conduct of Directors and Senior Management CSR Policy Whistle blower policy Jitsi Meet Mod Evasive Multisite using cPanel Softaculous Copying data from Linux to Windows using WinSCP Migrate MySQL Database Between DBaaS Manage MySQL Abuse Abuse Logs Abuse Attacks Spam Mails Phishing Complaint Copyright Complaint Billing Payment Payment Method EMandate Redeem Coupon Customer Details Updation Setup AutoDebit feature TDS Deduction process Declaration us 206AB Minimum Billing Provisioning and Deprovisioning Process View Invoice and Payment Prepaid Billing Set up Infra Credits Balance Alert Whatsapp notification for Payment reminder Restore Service on a Suspended Account for NonPayment Database Database Database PostgreSQL Parameter Group How to create Parameter Group MongoDB mariaDB Storage Storage Block Storage Object Storage CDP Backup Scalable File System Kubernetes Kubernetes Service Kubernetes Kubernetes Marketplace Monitoring Stack Cert Manager Nginx Ingress Controller Troubleshooting Argo CD Declarative GitOps CD for Kubernetes Deploy to Kubernetes using Argo CD and GitOps Jenkins CI Tool Setup Jenkins On Kubernetes Cluster Ansible CD Tool Istio Service Connecting to Dbaas FAQS Container Registery Container Registry Simplifying Pull Operations with UserFriendly Interface API Developers Guide API SDK E2E CLI Tool Introduction Terraform Terraform Getting Started e2enode Resource Node Plans e2eimage Resource AIML TIR AI Release Notes TIR AI Platform Introduction Getting Started Howto Guides Projects Notebooks Committed Notebook GPU H100 Plans Datasets Model Repository Model Endpoints Pipeline Run Scheduled Run Fine Tuning Models Model Playground Samples and Tutorials API Tokens SSH Key Container Team Features Settings Analytics Billing AIML Tutorials Finetune LLMA with Multiple GPUs Deploy Inference for LLMA 2 Deploy Inference for Codellama Deploy Inference for Stable Diffusion v21 Finetune Stable Diffusion model on TIR Deploy Inference for MPT7BCHAT Custom Containers in TIR Fine Tuning Bloom Natural Language Queries to SQL wth Code Llama E2E Networks documentation Welcome to TIR AI Platform Documentation Containers Containers TIR containers are fully collaborative environments that make AI development possible They combine power of containers jupyter labs and AIML frameworks to create a readily usable workspace for you and your entire team Some of the most common use cases are Run a script or container to finetune a Large Language Model LLM on single GPU using pytorch or huggingface train Run a script or container to tokenize and finetune LLMs or Diffusion models with with multiple GPUs single machine using deepspeed and accelerate Open and run a jupyter container ipynb from the platforms like github kaggle or collab Download and review datasets stored on TIR or other platforms like huggingface Download and test models like stable diffusion or any LLM Note A TIR container is fully functional coding environment If you prefer to work with command line shell over jupyter labs you can configure ssh on a container This way you can upload your data using sftp or sync your code with git tools and run the scripts as you would on your local system Getting Started Go to TIR Dashboard Create or Select a project Click on Container in sidebar section Click CREATE CONTAINER Choose an appropriate name for your container Select a container type as NEW NOTEBOOK If you want to open a container found on other platforms like github or google collab you can choose IMPORT NOTEBOOK option which allows entering the URL of the target container Next you can select from prebuilt or one of your own images For the sake of simplicity Click on Pytorch 2 option Next you can choose a CPU or GPU plan Feel free to choose Free Tier plan for this exercise Optional Set Enable SSH Access switch to enabled and add or select your SSH key Leave the rest of the options asis and click CREATE Next you will see a new container appear in the list of conatiner Wait for the container to come to a ready state When the container is ready you will see both Jupyter labs and SSH options if configured Choose any of these to access the container environment and work on your magic Container options TIR containers are extremely powerful and flexible While most configurations have a default to make our life eaiser sometimes you may need to tweak the knobs The following are the configurations that you can tweak in a container environments Enable SSH You can enable SSH access on the container using public key or password not recommended If you decide to enable ssh after starting a container you will have to first stop the container before you make changes Disk Size Each TIR container can have a disk size upto 1000GB The default is 30GB The selected disk will be mounted at homejovyan in your container environment We recommend using this path as your workspace so in case of restarts your content will be persistent Since TIR is containernative the changes that you make to any other paths on the container will not be persisted on restarts You can extend the disk size after the start of conatiner as well This workspace will be deleted when the associated container is deleted Note Please raise a support ticket if you need more than 1TB of disk workspace Local NVME Storage Only available for H100 plans This fast local storage will be available at mntlocal andonly for the duration of run We recommend using this path when you need faster writes eg save model checkpoints or reads Be sure to move this data to EOS bucket or under homejovyan before shutting down the container This type of storage is fixed and can not be expanded at anytime during the container cycle Plan Pricing You can choose between an hourly or committed plan We recommend using committed plans as they offer discounts and also may offer access to local NVME storage for H100 plans only Container Image TIR environments are containernative You can use prebuilt images with well known frameworks like pytorch tranformers or customise the prebuilt images You can make your own images TIRcompatible using image builder utility We recommend starting with prebuilt images In case you need to install packages from pip or aptget we recommend doing so from a jupyter notebook ipynb or maintaining requirementstxt Configuration TIR offers a variety of cpu and gpu options We recommend using A100 or H100 for best performance Update Container You can upgrade or downgrade both the configuration eg upgrade from cpu to gpu and Plan eg hourly to commited of a container if desired This is useful option when restarting containers and the original hardware plan gpu on the container is no longer available Stop Container If the plan and configuration allows you can stop a container and restart In case of hourly plan you will not be billed for the GPU or GPU when contanier is in a stopped state However if your disk usage is beyond free tier you will be charged for it Delete Container When a container is deleted all the resources associated with it will be deleted including the workspace disk Container Status Waiting The container instance is being deployed on the hardware of your choise Running The container is active and you can use either jupyter labs or ssh if enabled to access it Stopped The container is not assigned to any machine However the workspace disk mounted at homejovyan will continue to exist until you delete the container Depending on the size of the disk you will charged for the usage How to create Container To create a Container you have to click on Create Container which is at the right corner of the page After clicking on the Create Container button a page will appear now select container image option choose the Container Image and select the machine on which you want to run your service After that click on Next button After selecting the machine Resource page will appear now select the plan After that click on Next button After selecting the plan and click on next button Container details page will appear After entering all the details Disk Size Datasets and ssh key you will click on the create button for creating a container After clicking on the Create button the page will redirect to the Manage Containers page and display all details there After all steps completed The Container Summary details shown which is at the right corner of the page After clicking on the create button your GPU will be created and it will be shown like this Container Details Overview You can see the Container Details Plan Details under Overview tab Disk Size You can see the details disk size and also You can change the Disk size as per your requirements For updating the disk size you have to change the disk size and then click on update button Metrics You can see the Metrics graph in CPU Utilization Memory Utilization Interval You can see the one month activity as per your requirement in days hours Associated Datasets You can also see the Associated Datasets with two different datasets Mounted Unmounted You can also Unmount SSH Key You can see the SSH Key Details under ssh key tab Update SSH Key Note Only one SSH key can be added to a Launch Notebook from Sidebar Container Add SSH Key After Container Creation Note When user want to add ssh key after Container creation kindly first stop Container then add ssh key Container Actions You can see the actions like Launch Notebook Stop Update Container Delete Launch Notebook After clicking on Launch Notebook Notebook will be launched and it should be visible like this Stop Container For Stopping the container you have to click on Stop button and the Notebook will be stopped Update Container You can update Container For updating the Container You have to click on Update button Note Container must be in Stop state before updating the container Delete Container For Deleting the Container you have to click on Delete button On this page Getting Started Container options Container Status How to create Container Container Details Overview Disk Size Metrics Associated Datasets SSH Key Update SSH Key Container Add SSH Key After Container Creation Container Actions Launch Notebook Stop Container Update Container Delete Container Products TIR AI Platform GPU Dedicated Compute CPU Intensive Cloud High Memory Cloud Linux Smart Dedicated cPanel Linux Cloud Windows Cloud Windows SQL Cloud Plesk Windows Cloud GPU Smart Dedicated Load Balancer Company About Us Meet the Team Become a Partner E2E in Media Testimonials Investors Careers Contact Us Contact Sales Escalation Matrix Service Level Agreement Terms of Service Privacy Policy Refund Policy Policy FAQ Resources Blog Events Service Health Status Help White Papers Ecosystem Enablers Customers Certifications Countries Served FAQs E2E Networks Limited E2E Networks Limited is a NSE Listed AIFirst Hyperscale Cloud Computing Platform CIN Number L72900DL2009PLC341980 Copyright 2023 E2E Networks Limited


Refund Policy Claim your spot on the waitlist for the NVIDIA H100 GPUs Join WaitlistE2E CloudProductsPricingPartnersEventsCareersCompanyInvestorContact UsBlogLoginLoginSign UpRefund PolicyE2E Networks Limited E2E we our provides cloud platform and configuration services including but not limited to smart dedicated servers graphics processing units object storage content delivery network service and continuous data protection backup services Services to its customers Customers you your and such use of the Services by the Customers shall be governed by online terms of service available at link e2enetworkscompoliciestermsofservice andor a master services agreement if any executed between E2E and the Customer This refund policy Policy describes the cases in which a refund may be due to the Customer and sets out the terms governing such refund Your use of the Services or your registration with us for availing the Services constitutes your agreement to this Policy E2E may modify this Policy at any time by posting a revised version of the same on E2Es website and the amended version of this Policy shall become automatically binding on you if you continue to avail of the Services In connection of your use of our Services a refund may arise in the following casesA Refund for Invoice DiscrepanciesIf you find any discrepancies in our invoices with respect to usage metrics or the Services subscribed by you andor other details mentioned on the invoice like GSTIN etc you are required to email us at billingsupporte2enetworkscom within 7 Seven days of first receipt of Invoice on email id s registered with usInvoice EmailWe will verify the validity of the alleged discrepancies internally and after verifying whether any refund is due we may at our discretion offer you a credit note or infra credits andor refund the proportionate amount wherever requiredIf you do not inform us of any discrepancy within 7 Seven days of the date of the Invoice Email then thereafter we will not entertain any requests to modify billsoffer credit notes or offer refund in any other formB Refund in case of Deprovisioning of ServicesAll Services shall be presumed to be in an active state and shall be billed till the date of de provisioning of respective Services The Customer is required to follow the below process for making any deprovisioning requestServices availed through Self Service Portal The Services availed by you via self service portal accessible at httpsmyaccounte2enetworkscom can be deprovisioned by you by making a request through the selfservice portal For termination of the Services the Customer can follow the steps mentioned in help article available at link httpsdocse2enetworkscomnodesmanagenodehtmlServices Provisioned Manually Where Services have been provisioned manually by our team on your request you shall have to explicitly file a request for deprovisioning of the Services by sending an email to us at cloudplatforme2enetworkscom specifying explicitly the date of deprovisioning Such request shall be treated as valid only if sent through your authorized technical contact and the deprovisioning shall have to be unconditionalOn receiving your request for deprovisioning of Services our provisioning team will deprovision that those Services and send you and the E2E finance team a deprovisioning mail Deprovisioning Email We will cease billing for that those services from the effective de provisioning date mentioned in the DeProvisioning Email which is copied to our finance teamIn case our finance team inadvertently misses accounting for a Deprovisioning Email and has billed you for the deprovisioned services we shall rectify the error by issuing a credit note or infra credits on receipt of an email to that effect from you To enable us to do the same we request you to email us at billingsupporte2enetworkscom notifying us of the error along with the details of the appropriate Deprovisioning Email received from provisioninge2enetworkscom detailing the deprovisioning requestIn case a Deprovisioning Email has NOT been received by you we would request you to email cloudplatforme2enetworkscom intimating us of your previous deprovisioning request which was NOT processed along with a screenshot of such previous request The effective date of deprovisioning is usually the deprovisioning date mentioned in the Deprovisioning Email unless the service had a commitment period in which case the effective date would be the last date of such commitment periodIf you have NOT sent a deprovisioning request you should send it as soon as possible so that our team can proceed to issue a De provisioning Email to you Till the time the Services are de provisioned you remain liable to pay for the same The finance team will be able to issue credit notes or infra credits only as per the abovementioned processThe billing period for Services includes the day of provisioning and the day of deprovisioning and all the days in between these two dates All manually provisioned services including virtualdedicated compute nodes are billed in advance as per the monthly billing cycle unless you have specifically requested for a longer billing cycleIf you request us to deprovision any manually provisioned service before the end of its billing cycle we may at our discretion issue equivalent credit note or infra credits or refund the remaining amount charged in advance in full minus any payment gateway transaction charges to the source through which payment was received earlier Alternatively if you continue to avail other services that are still in the provisioned state the remaining amount may be adjusted in the next billing cycleC Refund for Software Licenses provided with the Compute NodesThe software licenses billed by us are billed for each calendar month regardless of the period of usage during the month Once a software license has been used during a calendar month it is NOT eligible for a refund and you will be charged for the full monthly cost of that license Refund of software license cost paid for unused future months is however possible on receipt of an email from you to us at billingsupporte2enetworkscom requesting the sameD Refund of Prepaid Infra CreditsIf you have any unused prepaid infra credits in your Customer Account and you dont intend to use any further Services then you can request refund by sending email to billingsupporte2enetworkscom provided Customer account is not in Inactive dormant stateFor the purpose of this clause Inactivedormant state shall mean a Customer account which at any point of time has not consumed any of the Services in the preceding 90 Ninety dayNo refund shall be due after the expiry of 90 day and we shall have the right to invalidate such credits after following the process mentioned in our TOSE TDS RefundIn order to claim a TDS refund please refer to our TDS help article accessible at TDS Refund How to claim TDS refunds E2E Networks Knowledgebase for information on the process to be followed When will you not be entitled to a refund under this PolicyYou shall not be entitled any refund in the following casesa Minimum Billing Committed Nodes The committed nodes are launched for a specific committed period for a fixed price and you shall not be eligible for a refund of the amount paid for the same Additionally you will not be eligible for a refund of the minimum billing amounts charged in each billing cycleb Promotional code benefits free coupons etc are not refundedc The transaction fees charged by payment gateways is not refundedd If you have an outstanding balance in any of the linked Customer accounts or if we suspect that your Customer account is linked in any manner to another customer accounts which has outstanding dues you will not be entitled to a refund The balance if any remaining after adjustment of the outstanding balance for such linked customer account shall be refunded to youFor the purpose of this clause outstanding dues shall include outstanding TDS payable for the Services where a duly signed TDS certificate has not been provided by youe In case of suspected violations of our TOS Company Policies as defined in our TOS etc orf Any other cases which at our discretion would disentitle you to a refund GeneralAll requests for refund shall be sent to billingsupporte2enetworkscom within the timeline mentioned herein above To the fullest extent permitted by applicable law you hereby waive all claims relating to feescharges payablealready paid to us unless claimed within the timeline prescribed in this refund policyProductsCPU Intensive CloudHigh Memory CloudLinux Smart DedicatedcPanel Linux CloudWindows CloudWindows SQL CloudPlesk Windows CloudGPU Smart DedicatedLoad BalancerCompanyAbout UsMeet the TeamBecome a PartnerE2E in MediaTestimonialsInvestorsCareersContact UsContact SalesEscalation MatrixService Level AgreementTerms of ServicePrivacy PolicyRefund PolicyPolicy FAQResourcesBlogEventsService Health StatusHelpWhitePapersEcosystem EnablersCustomersCertificationsCountries ServedFAQsE2E CloudE2E Networks Limited is Indias fastest growing accelerated cloud computing playerSubscribe to our newsletterGet latest news articles resources and trends delivered to your inbox weeklyThank you Your submission has been receivedOops Something went wrong while submitting the formCIN Number L72900DL2009PLC341980 Copyright 2024 E2E Networks Limited YouTubeInstagram


Custom Containers in TIR E2E Networks documentation Skip to content E2E Cloud Docs E2E Networks TIR Toggle navigation menu Login Sign Up E2E Networks documentation K Docs E2E Networks TIR Myaccount Getting Started with Myaccount SignUp Process for Indian Customers Customer validation Process for Indian Customers SignUp Process for International Customers Customer Validation Process for International Process Customer Validation Process for Contact Persons Domestic Customer Validation Process FAQs International Customer Validation Process FAQs Sign In Process Release Notes Compute Nodes Virtual Compute Nodes Monitoring 1Click Deployment Active Directory GPU GPU Cloud EQS Introduction Appliance Load Balancer Appliance Auto Scaling Introduction Concepts Define Scale Groups FaaS Function as a Service FaaS Network CDN VPC How to reserve an IP address Reserve IP Pool DNS Custom Reverse DNS Firewall Security Group Tag Security Firewall Bitninja Best Practices SSH Key Management Audit Logs Password Policy 2Factor Authentication Others License Management User Management Purchase SSL CertbotPlugin Apply or Redeem Coupon Code Account Statement Cloud Platform Support Introduction Platform Support Policy FAQ E2Es MyAccount FAQs Code of Conduct of Directors and Senior Management CSR Policy Whistle blower policy Jitsi Meet Mod Evasive Multisite using cPanel Softaculous Copying data from Linux to Windows using WinSCP Migrate MySQL Database Between DBaaS Manage MySQL Abuse Abuse Logs Abuse Attacks Spam Mails Phishing Complaint Copyright Complaint Billing Payment Payment Method EMandate Redeem Coupon Customer Details Updation Setup AutoDebit feature TDS Deduction process Declaration us 206AB Minimum Billing Provisioning and Deprovisioning Process View Invoice and Payment Prepaid Billing Set up Infra Credits Balance Alert Whatsapp notification for Payment reminder Restore Service on a Suspended Account for NonPayment Database Database Database PostgreSQL Parameter Group MongoDB mariaDB Storage Storage Block Storage Object Storage CDP Backup Scalable File System Kubernetes Kubernetes Service Kubernetes Kubernetes Marketplace Monitoring Stack Cert Manager Nginx Ingress Controller Troubleshooting Argo CD Declarative GitOps CD for Kubernetes Deploy to Kubernetes using Argo CD and GitOps Jenkins CI Tool Setup Jenkins On Kubernetes Cluster Ansible CD Tool Istio Service Connecting to Dbaas FAQS Container Registery Container Registery API Developers Guide API SDK E2E CLI Tool Introduction Terraform Terraform Getting Started e2enode Resource Node Plans e2eimage Resource AIML TIR AI Release Notes TIR AI Platform Introduction Getting Started Howto Guides Projects Notebooks Committed Notebook GPU H100 Plans Datasets Model Repository Model Endpoints Pipeline Run Scheduled Run Fine Tuning Models Model Playground Samples and Tutorials API Tokens SSH Key Container Team Features Settings Analytics Billing AIML Tutorials Finetune LLMA with Multiple GPUs Deploy Inference for LLMA 2 Deploy Inference for Codellama Deploy Inference for Stable Diffusion v21 Finetune Stable Diffusion model on TIR Deploy Inference for MPT7BCHAT Custom Containers in TIR Fine Tuning Bloom Natural Language Queries to SQL wth Code Llama E2E Networks documentation Welcome to TIR AI Platform Documentation Tutorials Custom Containers in TIR Custom Containers in TIR TIR platform supports a variety of prebuilt containers that can launch API handlers for you But sometimes you may want to handle the API requests differently or introduce additional steps in the flow This is when a custom container image can help Additionally you may also have your own containers that you want to launch with a GPU plan In this tutorial we will Write an API handler to handle model inference requests Package the API handler in a container image Configure a model endpoint in TIR to serve the model over REST API Use TIR Models to improve launch time of containers Step 1 Write a API handler for model inference By default each Model Endpoint in TIR follows Kserve Open inference protocol for handling inference requests We recommend using the same format for your REST API endpoints but you may choose to do things differently In this tutorial we will use Kserve Model Server to wrap our model inference calls so we dont have to deal with liveness and readiness probes Lets walk through a simple template of API handler If you intend to use kserve model server your code must include methods such as load predict and extend kserveModel as shown below from kserve import Model ModelServer class MyCustomModelModel def initself name str superinitname selfname name selfready False selfload def loadself fetch your model from disk or remote selfmodel def predictself payload Dict headers Dictstr str None Dict read request input from payload dict for example inputs payloadinstances sourcetext inputs0text call inference result return predictions result if name main here we have named the model metallama27bchat but you may choose any name of your choice This is important because it impacts your REST endpoint lets say you define a model name as mnist then your rest endpoints will end with httpsinfere2enetworksnetprojectprojectidendpointisendpointidv1modelsmnist model MyCustomModelmetallama27bchat ModelServerstartmodel To take this further create a project directory on your local or TIR notebook and create modelserverpy with the following contents filename modelserverpy from kserve import Model ModelServer from transformers import AutoTokenizer AutoModelForCausalLM import transformers import torch from typing import List Dict class MetaLLMA2ModelModel def initself name str superinitname selfname name selfready False selftokenizer None selfmodelid metallamaLlama27bchathf selfload def loadself this step fetches the model from huggingface directly the downloads may take longer and be slow depending on upstream link We recommend using TIR Models instead selfmodel AutoModelForCausalLMfrompretrainedselfmodelid trustremotecodeTrue devicemapauto selftokenizer AutoTokenizerfrompretrainedselfmodelid selfpipeline transformerspipeline textgeneration modelselfmodel torchdtypetorchfloat16 tokenizerselftokenizer devicemapauto selfready True def predictself payload Dict headers Dictstr str None Dict inputs payloadinstances sourcetext inputs0text sequences selfpipelinesourcetext dosampleTrue topk10 numreturnsequences1 eostokenidselftokenizereostokenid maxlength200 results for seq in sequences resultsappendseqgeneratedtext return predictions results if name main model MetaLLMA2Modelmetallama27bchat ModelServerstartmodel Note The LLMA 2 model weights need to be downloaded from huggingface following the licensing terms Once you have the weights on your local machine or TIR notebook you can upload them to Model bucket in EOS Step 2 Package the API handler in a container image Now Lets package our API handler from step 1 using the docker file below Dockerfile FROM pytorchtorchservekfs081gpu ENV APPHOME app WORKDIR APPHOME Install production dependencies COPY requirementstxt RUN pip install nocachedir r requirementstxt Copy local code to container image COPY modelserverpy CMD python modelserverpy We need to create and push the container image to docker hub now You may also choose to use your private repo docker build t yourdockerhandleheremetallm2server docker push yourdockerhandleheremetallm2server Note You may run the docker to test the API provided your hardware can support the LLMA2 model If you are on TIR Notebook with A10080 or your local machine can support the model then do go ahead and test the api locally Step 3 Configure a model endpoint in TIR to serve the model over REST API Now have a container image in docker hub we can define a model endpoint in TIR Go to TIR Dashboard Select a Project Go to Model Endpoints Click Create Endpoint Select Custom Container and Press Continue Select a GPU plan GDC3A10080 Set Disk Size to 15G or higher depending on the model size Click Next Enter an appropriate name for the endpoint Click Next In Container Details enter image as yourdockerhandleheremetallm2server and select other parameters as necessary In Environment details enter these keyvals HUGGINGFACEHUBTOKEN get the token from huggingface website TRANSFORMERSCACHE mntmodels In Model Details Do not select a model In above example we are fetching the model from huggingface directly so we dont need to fetch model from EOS Click Finish and create the endpoint If all goes well you will see the endpoint come to a ready state When it does you can test the model using the curl commands from sample API request tab Sample API request to see readiness of the endpoint curl H Authorization Bearer token httpsinfere2enetworksnetprojectprojectendpointendpointidv1modelsmetallama27bchat If the model is ready you will see a response Responsename metallama27bchat ready truefalse Sample API request to test the model code Request formatinstances Response formatpredictions curl H Authorization Bearer token X POST httpsinfere2enetworksnetprojectprojectendpointendpointidv1modelsmetallama27bchatpredict d instancestext Life is such that Step 4 Use TIR Models to improve launch time of containers You will notice that the model endpoints takes a while to be deployed or may time out as well in some cases This is because our modelserverpy is trying to download the model directly from huggingface hub To fix this we can define a TIR Model and host the model weights in in EOS bucket Go to TIR Dashboard Go to Models Create a new Model with name eg mymodel with format custom Once the TIR model is created you will get EOS bucket details Use the instructions from Setup Minio CLI tab to configure Minio Host on your local or TIR Notebook Download the target model eg metallamaLlama27bchathf from huggingface hub Upload the model code and weightsfrom HOMEcachehuggingfacehubmodelsnapshot directory to EOS bucket using minio cp command You can use the cp command template from Setup Minio CLI tab Now go ahead with the step 3 above but this time choose model eg mymodel in model details section The endpoint created will now ensure that the model weights are downloaded to mntmodels directory before starting the API handler You may need to also change the modelserverpy to load weights from mntmodels and not huggingface hub Products TIR AI Platform GPU Dedicated Compute CPU Intensive Cloud High Memory Cloud Linux Smart Dedicated cPanel Linux Cloud Windows Cloud Windows SQL Cloud Plesk Windows Cloud GPU Smart Dedicated Load Balancer Company About Us Meet the Team Become a Partner E2E in Media Testimonials Investors Careers Contact Us Contact Sales Escalation Matrix Service Level Agreement Terms of Service Privacy Policy Refund Policy Policy FAQ Resources Blog Events Service Health Status Help White Papers Ecosystem Enablers Customers Certifications Countries Served FAQs E2E Networks Limited E2E Networks Limited is a NSE Listed AIFirst Hyperscale Cloud Computing Platform CIN Number L72900DL2009PLC341980 Copyright 2023 E2E Networks Limited