Package: cofutils
Introduction for Cof utilsThere're several useful tools for experiments, such as cofrun, coftimer, cofmem, cofwriter.
The Overview ofCofutilsInstallBy Pypipip install cofutilsBy Sourcegit clone https://gitee.com/haiqwa/cofutils.git
pip install .UsageCof WriterCof LoggerCof logger can print user message according to print-level.
In *.py:from cofutils import coflogger
coflogger.debug("this is debug")
coflogger.info("this is info")
coflogger.warn("this is warn")
coflogger.error("this is error")Print-level is determined by environment variableCOF_DEBUG:COF_DEBUG=WARN python main.pyThe default print-level isINFO. By the way, only the node of 'rank=0' can output log in distributed environmentCof CSVDump data into csv format.Get a unique csv writer by calling cofcsvWrite data in dict type. You can append data at anywhere and anytimeSave data as[name].csvunder theroot_dir. After that cofcsv will clear data in defaultfromcofutilsimportcofcsvdata={'a':1,'b':2,'c':3}test_csv=cofcsv('test')test_csv.write(data)data={'a':4,'b':5,'c':6}test_csv.write(data)# remember to save data by calling cofcsv.savecofcsv.save(root_dir='csv_output')Cof TbWrite data into tensorboard.fromcofutilsimportcoftbcoftb('test')coftb.write({'a':10})coftb.write({'a':20})coftb.write({'a':30})coftb.close()By default,events.out.tfevents.xxxwould be dump tocoftbdirectory.tensorboard--logdircoftb/Cof TimerCof timer is similar to theTimerinMegatron-LM. By default, the timer achieves the duration time of operations on the host side. If you want to profile cuda programme, please setcuda_timer=True, which obtains execution time by cuda events.It support two log modes which can be set by the keywordtimedict:Organize the result into a string and output it intoSTDOUTwhich is easy to view for usersDirectly return the result time table asDictformatUsers can also customize their time log writer by settingwriter. Currently, cof timer supportscsv,tb,info,debug,warn,erroras writer function.Note: if you call.logto print time, thenthe timer will reset automaticallyfromcofutilsimportcoftimer,coflogger,coftb,cofcsvimporttimeimporttorchcoftimer.set_writer(writer="warn,csv,tb",name="loop_sleep")test_1=coftimer('test1')test_2=coftimer('test2')test_3=coftimer('test3',cuda_timer=True)for_inrange(3):test_1.start()time.sleep(1)test_1.stop()coftimer.log(normalizer=3,timedict=False)withtest_2:for_inrange(3):time.sleep(1)coftimer.log(normalizer=3,timedict=False)m1=torch.randn(1024,1024,16,device="cuda:0")m2=torch.randn(1024,1024,16,device="cuda:0")withtest_3:for_inrange(3):m1=m1+m2m1.div_(20)m2.div_(10)time_dict=coftimer.log(normalizer=3,timedict=True)coflogger.info(time_dict)cofcsv.save()[2023-11-2123:08:33.670][CofINFO]:time(ms)|test1:1001.15|test2:0.00[2023-11-2123:08:36.674][CofWARNING]:time(ms)|test1:1001.11|test2:0.00[2023-11-2123:08:39.678][CofINFO]:{'test1':0.0,'test2':1001.1359850565592}Cof Memory ReportPrint GPU memory states by pytorch cuda API. And it supports to dump memory states into tensorboard of csv, except for printing out to the terminal.MA: memory current allocatedMM: max memory allocatedMR: memory reserved by pytorchcofmemis a time-cost API. Please remember to remove it if you want to profiling the performance of program. Similarly, you can set writer forcofmem.cofmemwould do nothing if you set writer asNone.The latency ofcofmem:writerlatencyNone0mslogger.info0.8mstensorboard2.8mscsv0.5msfromcofutilsimportcofmem,cofcsv,coftimerimporttorchcofmem.set_writer('tb,csv',name="test-1")coftimer.set_writer('tb,csv',name="test-1")timer=coftimer(name='test-1')cofmem("Before Init Random Tensor")tensor1=torch.rand((1024,1024,128),dtype=torch.float32,device='cuda:0')tensor2=torch.rand((1024,1024,128),dtype=torch.float32,device='cuda:0')withtimer:cofmem("After Init Random Tensor")add_result=tensor1+tensor2cofmem("After Addition")subtract_result=tensor1-tensor2cofmem("After Subtraction")multiply_result=tensor1*tensor2cofmem("After Multiplication")divide_result=tensor1/tensor2cofmem("After Division")coftimer.log()cofcsv.save()Note thatcofmemwould return a dict which contains memory report.(deepspeed)haiqwa@gpu9:~/documents/cofutils$python~/test.py[2023-11-1115:32:46.873][CofINFO]:beforexxxGPUMemoryReport(GB):MA=0.00|MM=0.00|MR=0.00[2023-11-1115:32:46.873][CofINFO]:afterxxxGPUMemoryReport(GB):MA=0.00|MM=0.00|MR=0.00Cofrun is all you need!User can easily launch distributed task bycofrun. What users need to do is to provide a template bash file and configuration json file.You can see the examples inexample/usage: cofrun [-h] [--file FILE] [--input INPUT] [--template TEMPLATE] [--output OUTPUT] [--test] [--nsys] [--list] [--range RANGE]

optional arguments:
  -h, --help            show this help message and exit
  --file FILE, -f FILE  config file path, default is ./config-template.json
  --input INPUT, -i INPUT
                        run experiments in batch mode. all config files are placed in input directory
  --template TEMPLATE, -T TEMPLATE
                        provide the path of template .sh file
  --output OUTPUT, -o OUTPUT
                        write execution output to specific path
  --test, -t            use cof run in test mode -> just generate bash script
  --nsys, -n            use nsys to profile your cuda programme
  --list, -l            list id of all input files, only available when input dir is provided
  --range RANGE, -r RANGE
                        support 3 formats: [int | int,int,int... | int-int], and int value must be > 0Let's run the example:cofrun -f demo_config.json -T demo_template.shAnd the execution history of cofrun will be written intohistory.cof