Package: calflops
calflops: a FLOPs and Params calculate tool for neural networksEnglish|中文IntroductionThis tool(calflops) is designed to compute the theoretical amount of FLOPs(floating-point operations)、MACs(multiply-add operations) and Parameters in all various neural networks, such as Linear、 CNN、 RNN、 GCN、Transformer(Bert、LlaMA etc Large Language Model)，even includingany custom modelsviatorch.nn.function.*as long as based on the Pytorch implementation. Meanwhile this tool supports the printing of FLOPS, Parameter calculation value and proportion of each submodule of the model, it is convient for users to understand the performance consumption of each part of the model.For LLM, this is probably the easiest tool to calculate FLOPs and it is very convenient forhuggingfaceplatform models. You can usecalflops.calculate_flops_hf(model_name)bymodel_namewhich inhuggingface modelsto calculate model FLOPs without downloading entire model weights locally.Notice this method requires the model to support the empty model being created for model inference in meta device.fromcalflopsimportcalculate_flops_hfmodel_name="meta-llama/Llama-2-7b"access_token="..."# your application token for using llama2flops,macs,params=calculate_flops_hf(model_name=model_name,access_token=access_token)# default input shape: (1, 128)print("%sFLOPs:%sMACs:%sParams:%s\n"%(model_name,flops,macs,params))If model can't inference in meta device,  you just need assign llm corresponding tokenizer to the parameter:transformers_tokenizerto pass in funcional ofcalflops.calculate_flops(), and it will automatically help you build the model input data whose size is input_shape.  Alternatively, you also can pass in the input data of models which need multi data as input that you have constructed.In addition, the implementation process of this package inspired byptflops、deepspeed、hf acceleratelibraries, Thanks for their great efforts, they are both very good work. Meanwhile this package also improves some aspects to calculate FLOPs based on them.How to installInstall the latest versionFrom PyPI:pipinstallcalflopsAnd you also can download latestcalflops-*-py3-none-any.whlfiles fromhttps://pypi.org/project/calflops/pipinstallcalflops-*-py3-none-any.whlHow to use calflopsExampleCNN ModelIf model has only one input, you just need set the model input size by parameterinput_shape, it can automatically generate random model input to complete the calculation:fromcalflopsimportcalculate_flopsfromtorchvisionimportmodelsmodel=models.alexnet()batch_size=1input_shape=(batch_size,3,224,224)flops,macs,params=calculate_flops(model=model,input_shape=input_shape,output_as_string=True,output_precision=4)print("Alexnet FLOPs:%sMACs:%sParams:%s\n"%(flops,macs,params))#Alexnet FLOPs:4.2892 GFLOPS   MACs:2.1426 GMACs   Params:61.1008 MIf the model has multiple inputs, use the parametersargsorkargs, as shown in the Transfomer Model below.Calculate Huggingface Model By Model NameNo need to download the entire parameter weight of the model to the local, just by the model name can test any open source large model on the huggingface platform.fromcalflopsimportcalculate_flops_hfbatch_size,max_seq_length=1,128model_name="baichuan-inc/Baichuan-13B-Chat"flops,macs,params=calculate_flops_hf(model_name=model_name,input_shape=(batch_size,max_seq_length))print("%sFLOPs:%sMACs:%sParams:%s\n"%(model_name,flops,macs,params))You can also use this model urls of huggingface platform to calculate it FLOPs.fromcalflopsimportcalculate_flops_hfbatch_size,max_seq_length=1,128model_name="https://huggingface.co/THUDM/chatglm2-6b"# THUDM/chatglm2-6bflops,macs,params=calculate_flops_hf(model_name=model_name,input_shape=(batch_size,max_seq_length))print("%sFLOPs:%sMACs:%sParams:%s\n"%(model_name,flops,macs,params))There are some model uses that require an application first, and you only need to pass the application in through theaccess_tokento calculate its FLOPs.fromcalflopsimportcalculate_flops_hfbatch_size,max_seq_length=1,128model_name="meta-llama/Llama-2-7b"access_token=""# your application token for using llama2flops,macs,params=calculate_flops_hf(model_name=model_name,access_token=access_token,input_shape=(batch_size,max_seq_length))print("%sFLOPs:%sMACs:%sParams:%s\n"%(model_name,flops,macs,params))Transformer Model (Local)Compared to the CNN Model, Transformer Model if you want to use the parameterinput_shapeto make calflops automatically generating the input data, you should pass its corresponding tokenizer through the parametertransformer_tokenizer.# Transformers Model, such as bert.fromcalflopsimportcalculate_flopsfromtransformersimportAutoModelfromtransformersimportAutoTokenizerbatch_size,max_seq_length=1,128model_name="hfl/chinese-roberta-wwm-ext/"model_save="../pretrain_models/"+model_namemodel=AutoModel.from_pretrained(model_save)tokenizer=AutoTokenizer.from_pretrained(model_save)flops,macs,params=calculate_flops(model=model,input_shape=(batch_size,max_seq_length),transformer_tokenizer=tokenizer)print("Bert(hfl/chinese-roberta-wwm-ext) FLOPs:%sMACs:%sParams:%s\n"%(flops,macs,params))#Bert(hfl/chinese-roberta-wwm-ext) FLOPs:67.1 GFLOPS   MACs:33.52 GMACs   Params:102.27 MIf you want to use your own generated specific data to calculate FLOPs, you can use
parameterargsorkwargs，and parameterinput_shapecan no longer be assigned to pass in this case. Here is an example that can be seen is inconvenient comparedt to use parametertransformer_tokenizer.# Transformers Model, such as bert.fromcalflopsimportcalculate_flopsfromtransformersimportAutoModelfromtransformersimportAutoTokenizerbatch_size,max_seq_length=1,128model_name="hfl/chinese-roberta-wwm-ext/"model_save="/code/yexiaoju/generate_tags/models/pretrain_models/"+model_namemodel=AutoModel.from_pretrained(model_save)tokenizer=AutoTokenizer.from_pretrained(model_save)text=""inputs=tokenizer(text,add_special_tokens=True,return_attention_mask=True,padding=True,truncation="longest_first",max_length=max_seq_length)iflen(inputs["input_ids"])<max_seq_length:apply_num=max_seq_length-len(inputs["input_ids"])inputs["input_ids"].extend([0]*apply_num)inputs["token_type_ids"].extend([0]*apply_num)inputs["attention_mask"].extend([0]*apply_num)inputs["input_ids"]=torch.tensor([inputs["input_ids"]])inputs["token_type_ids"]=torch.tensor([inputs["token_type_ids"]])inputs["attention_mask"]=torch.tensor([inputs["attention_mask"]])flops,macs,params=calculate_flops(model=model,kwargs=inputs,print_results=False)print("Bert(hfl/chinese-roberta-wwm-ext) FLOPs:%sMACs:%sParams:%s\n"%(flops,macs,params))#Bert(hfl/chinese-roberta-wwm-ext) FLOPs:22.36 GFLOPS   MACs:11.17 GMACs   Params:102.27 MLarge Language ModelOnlinefromcalflopsimportcalculate_flops_hfbatch_size,max_seq_length=1,128model_name="meta-llama/Llama-2-7b"access_token=""# your application for using llamaflops,macs,params=calculate_flops_hf(model_name=model_name,access_token=access_token,input_shape=(batch_size,max_seq_length))print("%sFLOPs:%sMACs:%sParams:%s\n"%(model_name,flops,macs,params))LocalNote here that the tokenizer must correspond to the llm model because llm tokenizer processes maybe are different.#Large Languase Model, such as llama2-7b.fromcalflopsimportcalculate_flopsfromtransformersimportLlamaTokenizerfromtransformersimportLlamaForCausalLMbatch_size,max_seq_length=1,128model_name="llama2_hf_7B"model_save="../model/"+model_namemodel=LlamaForCausalLM.from_pretrained(model_save)tokenizer=LlamaTokenizer.from_pretrained(model_save)flops,macs,params=calculate_flops(model=model,input_shape=(batch_size,max_seq_length),transformer_tokenizer=tokenizer)print("Llama2(7B) FLOPs:%sMACs:%sParams:%s\n"%(flops,macs,params))#Llama2(7B) FLOPs:1.7 TFLOPS   MACs:850.00 GMACs   Params:6.74 BShow each submodule result of FLOPs、MACs、ParamsThe calflops provides a more detailed display of model FLOPs calculation information. By setting the parameterprint_result=True, which defaults to True, flops of the model will be printed in the terminal or jupyter interface.Meanwhile, by setting the parameterprint_detailed=Truewhich default to True, the calflops supports the display of the calculation results and proportion of FLOPs、MACs and Parameter in each submodule of the entire model, so that it is convenient to see the largest part of the calculation consumption in the entire model.More use introductionHow to make output format more elegantYou can use parameter output_as_string、output_precision、output_unit to determine the format of output data is value or string, if it is string, how many bits of precision to retain and the unit of value, such as FLOPs, the unit of result is "TFLOPs" or "GFLOPs", "MFLOPs".How do deal with model has multiple inputsThe calflops support  multiple inputs of model, just use parameter args or kwargs to construct multiple inputs can be passed in as model inference.How to calculate the results of FLOPS include forward and backpropagation of the modelYou can use the parameter include_backPropagation to select whether the calculation of FLOPs results includes the process of model backpropagation. The default is False, that is result of FLOPs only include forward.In addition, the parameter compute_bp_factor to determine how many times backward as much computation as forward propagation.The defaults that is 2.0, according tohttps://epochai.org/blog/backward-forward-FLOP-ratioHow to calculate FLOPs for only part of the model moduleYou can use the parameter ignore_modules to select which modules of model are ignored during FLOPs calculation. The default is [], that is all modules of model would be calculated in results.How to calculate FLOPs of the generate function in LLMYou just need to assign "generate" to parameter forward_mode.APIof thecalflopscalflops.calculate_flops()fromcalflopsimportcalculate_flopsdefcalculate_flops(model,input_shape=None,transformer_tokenizer=None,args=[],kwargs={},forward_mode="forward",include_backPropagation=False,compute_bp_factor=2.0,print_results=True,print_detailed=True,output_as_string=True,output_precision=2,output_unit=None,ignore_modules=None):"""Returns the total floating-point operations, MACs, and parameters of a model.Args:model ([torch.nn.Module]): The model of input must be a PyTorch model.input_shape (tuple, optional): Input shape to the model. If args and kwargs is empty, the model takes a tensor with this shape as the only positional argument. Default to [].transformers_tokenizer (None, optional): Transforemrs Toekenizer must be special if model type is transformers and args、kwargs is empty. Default to Noneargs (list, optinal): list of positional arguments to the model, such as bert input args is [input_ids, token_type_ids, attention_mask]. Default to []kwargs (dict, optional): dictionary of keyword arguments to the model, such as bert input kwargs is {'input_ids': ..., 'token_type_ids':..., 'attention_mask':...}. Default to{}forward_mode (str, optional): To determine the mode of model inference, Default to 'forward'. And use 'generate' if model inference uses model.generate().include_backPropagation (bool, optional): Decides whether the final return FLOPs computation includes the computation for backpropagation.compute_bp_factor (float, optional): The model backpropagation is a multiple of the forward propagation computation. Default to 2.print_results (bool, optional): Whether to print the model profile. Defaults to True.print_detailed (bool, optional): Whether to print the detailed model profile. Defaults to True.output_as_string (bool, optional): Whether to print the output as string. Defaults to True.output_precision (int, optional) : Output holds the number of decimal places if output_as_string is True. Default to 2.output_unit (str, optional): The unit used to output the result value, such as T, G, M, and K. Default is None, that is the unit of the output decide on value.ignore_modules ([type], optional): the list of modules to ignore during profiling. Defaults to None.calflops.calculate_flops_hf()defcalculate_flops_hf(model_name,input_shape=None,library_name="transformers",trust_remote_code=True,access_token="",forward_mode="forward",include_backPropagation=False,compute_bp_factor=2.0,print_results=True,print_detailed=True,output_as_string=True,output_precision=2,output_unit=None,ignore_modules=None):"""Returns the total floating-point operations, MACs, and parameters of a model.Args:model_name (str): The model name in huggingface platform https://huggingface.co/models, such as meta-llama/Llama-2-7b、baichuan-inc/Baichuan-13B-Chat etc.input_shape (tuple, optional): Input shape to the model. If args and kwargs is empty, the model takes a tensor with this shape as the only positional argument. Default to [].trust_remote_code (bool, optional): Trust the code in the remote library for the model structure.access_token (str, optional): Some models need to apply for a access token, such as meta llama2 etc.forward_mode (str, optional): To determine the mode of model inference, Default to 'forward'. And use 'generate' if model inference uses model.generate().include_backPropagation (bool, optional): Decides whether the final return FLOPs computation includes the computation for backpropagation.compute_bp_factor (float, optional): The model backpropagation is a multiple of the forward propagation computation. Default to 2.print_results (bool, optional): Whether to print the model profile. Defaults to True.print_detailed (bool, optional): Whether to print the detailed model profile. Defaults to True.output_as_string (bool, optional): Whether to print the output as string. Defaults to True.output_precision (int, optional) : Output holds the number of decimal places if output_as_string is True. Default to 2.output_unit (str, optional): The unit used to output the result value, such as T, G, M, and K. Default is None, that is the unit of the output decide on value.ignore_modules ([type], optional): the list of modules to ignore during profiling. Defaults to None.Example:.. code-block:: pythonfrom calflops import calculate_flops_hfbatch_size = 1max_seq_length = 128model_name = "baichuan-inc/Baichuan-13B-Chat"flops, macs, params = calculate_flops_hf(model_name=model_name,input_shape=(batch_size, max_seq_length))print("%s FLOPs:%s  MACs:%s  Params:%s \n" %(model_name, flops, macs, params))Returns:The number of floating-point operations, multiply-accumulate operations (MACs), and parameters in the model."""calflops.generate_transformer_input()defgenerate_transformer_input(model_tokenizer,input_shape,device):"""Automatically generates data in the form of transformes model input format.Args:input_shape (tuple):transformers model input shape: (batch_size, seq_len).tokenizer (transformer.model.tokenization): transformers model tokenization.tokenizer.Returns:dict: data format of transformers model input, it is a dict which contain 'input_ids', 'attention_mask', 'token_type_ids' etc."""Citationif calflops was useful for your paper or tech report, please cite me:@online{calflops,
  author = {xiaoju ye},
  title = {calflops: a FLOPs and Params calculate tool for neural networks in pytorch framework},
  year = 2023,
  url = {https://github.com/MrYxJ/calculate-flops.pytorch},
}Common model calculate flopsLarge Language ModelInput data format: batch_size=1, seq_len=128fwd FLOPs: The FLOPs of the model forward propagationbwd + fwd FLOPs: The FLOPs of model forward and backward propagationIn addition, note that fwd + bwd does not include calculation losses for model parameter  activation recomputation, if the results wants to include activation recomputation that is only necessary to multiply the result of fwd FLOPs by 4(According to the paper:https://arxiv.org/pdf/2205.05198.pdf), and in calflops you can easily set parametercomputer_bp_factor=3to make the result of including the activate the recalculation.ModelInput ShapeParams(B)Params(Total)fwd FLOPs(G)fwd MACs(G)fwd + bwd FLOPs(G)fwd + bwd MACs(G)bloom-1b7(1,128)1.72B1722408960310.92155.42932.76466.27bloom-7b1(1,128)7.07B70690160641550.39775.114651.182325.32bloomz-1b7(1,128)1.72B1722408960310.92155.42932.76466.27baichuan-7B(1,128)7B70005596161733.62866.785200.852600.33chatglm-6b(1,128)6.17B61732864001587.66793.754762.972381.24chatglm2-6b(1,128)6.24B62435840001537.68768.84613.032306.4Qwen-7B(1,128)7.72B77213245441825.83912.885477.482738.65llama-7b(1,128)6.74B67384156161700.068505100.192550llama2-7b(1,128)6.74B67384156161700.068505100.192550llama2-7b-chat(1,128)6.74B67384156161700.068505100.192550chinese-llama-7b(1,128)6.89B68854865921718.89859.415156.672578.24chinese-llama-plus-7b(1,128)6.89B68854865921718.89859.415156.672578.24EleutherAI/pythia-1.4b(1,128)1.31B1311625216312.54156.23937.61468.69EleutherAI/pythia-12b(1,128)11.59B115865497602911.471455.598734,414366.77moss-moon-003-sft(1,128)16.72B167179801604124.932062.3912374.86187.17moss-moon-003-sft-plugin(1,128)16.06B160604160003956.621978.2411869.95934.71We can draw some simple and interesting conclusions from the table above:The chatglm2-6b in the model of the same scale, the model parameters are smaller, and FLOPs is also smaller, which has certain advantages in speed performance.The parameters of the llama1-7b, llama2-7b, and llama2-7b-chat models did not change at all, and FLOPs remained consistent. The structure of the model that conforms to the 7b described bymeta in its llama2 reporthas not changed, the main difference is the increase of training data tokens.Similarly, it can be seen from the table that the chinese-llama-7b and chinese-llama-plus-7b data are also in line withcui's report, just more chinese data tokens are added for training, and the model structure and parameters do not change.......More model FLOPs would be updated successively, see githubcalculate-flops.pytorchBertInput data format: batch_size=1, seq_len=128ModelInput ShapeParams(M)Params(Total)fwd FLOPs(G)fwd MACs(G)fwd + bwd FLOPs()fwd + bwd MACs(G)hfl/chinese-roberta-wwm-ext(1,128)102.27M10226764867.133.52201.3100.57......You can use calflops to calculate the more different transformer models based bert, look forward to updating in this form.BenchmarktorchvisionInput data format: batch_size = 1, actually input_shape = (1, 3, 224, 224)Note: The FLOPs in the table only takes into account the computation of forward propagation of the model,Totalrefers to the total numerical representation without unit abbreviations.ModelInput ResolutionParams(M)Params(Total)FLOPs(G)FLOPs(Total)Macs(G)Macs(Total)alexnet224x22461.10611008401.431429740000741.197418800000vgg11224x224132.8613286300015.24152392000007.617609090000vgg13224x224133.0513304800022.652264760000011.3111308500000vgg16224x224138.3613835800030.973097380000015.4715470300000vgg19224x224143.6714366700039.303930000000019.6319632100000vgg11_bn224x224132.8713286900015.25152540000007.617609090000vgg13_bn224x224133.0513305400022.672267210000011.3111308500000vgg16_bn224x224138.3713836600031.003100090000015.4715470300000vgg19_bn224x224143.6814367800039.333932970000019.6319632100000resnet18224x22411.69116895003.6436362500001.811814070000resnet34224x22421.80217977007.3473393900003.663663760000resnet50224x22425.56255570008.2182111100004.094089180000resnet101224x22444.554454920015.65156909000007.807801410000resnet152224x22460.196019280023.092309430000011.5111513600000squeezenet1_0224x2241.2512484201.6516489700000.82818925000squeezenet1_1224x2241.2412355000.717050140000.35349152000densenet121224x2247.9879788605.7257168800002.832834160000densenet169224x22414.15141950006.7867783700003.363359840000densenet201224x22420.01200139008.6686585200004.294291370000densenet161224x22428.682868100015.5515546500007.737727900000inception_v3224x22427.16271613005.2956923900002.842837920000Thanks to @zigangzhao-aiusecalflopsto static torchvision form.You also can compare torchvision results of calculate FLOPs with anthoer good tool:ptflops readme.md.Concact AuthorAuthor:MrYXJMail:yxj2017@gmail.com