Package: pix2text
👩🏻‍💻 Online Demo|💬 Contact中文| EnglishPix2TextUpdate 2024.02.26:V1.0ReleasedMain Changes:The Mathematical Formula Recognition (MFR) model employs a new architecture and has been trained on a new dataset, achieving state-of-the-art (SOTA) accuracy. For detailed information, please see:Pix2Text V1.0 New Release: The Best Open-Source Formula Recognition Model | Breezedeus.com.Update 2024.01.10:V0.3ReleasedMajor Changes:Support for recognizing80+languages; for a detailed list of supported languages, seeList of Supported Languages;Added domestic sites for automatic model downloads;Optimized the logic for merging detection boxes.Update 2023.07.03: V0.2.3 ReleasedMajor changes:Trained a newformula recognition modelforP2T Online Serviceto use. The new model has higher accuracy, especially forhandwritten formulasandmulti-line formulas. See:New Formula Recognition Model for Pix2Text | Breezedeus.com.Optimized the sorting logic of detected boxes and the processing logic of mixed images to make the final recognition results more intuitive.Optimized the merging logic of recognition results to automatically determine line breaks and paragraph breaks.See more at:RELEASE.md.Pix2Text (P2T)aims to be afree and open-source Pythonalternative toMathpix. It can already complete the core functionalities ofMathpix. Starting fromV0.2,Pix2Text (P2T)supports recognizingmixed images containing both text and formulas, with output similar toMathpix. The core principles of P2T are shown below (text recognition supports bothChineseandEnglish):P2Tutilizes the open-source toolCnSTDto detect the locations ofmathematical formulasin images. These detected areas are then processed byP2T's ownformula recognition engine (LatexOCR)to recognize the LaTeX representation of each mathematical formula. The remaining parts of the image are processed by atext recognition engine (CnOCRorEasyOCR)for text detection and recognition. Finally,P2Tmerges all recognition results to obtain the final image recognition outcome. Thanks to these great open-source projects!For beginners who are not familiar with Python, we also provide thefree-to-useP2T Online Service. Just upload your image and it will output the P2T parsing results.The online service uses the latest models and works better than the open-source ones.If interested, please scan the QR code below to add the assistant WeChat account, and sendp2tto get invited to the P2T user group. The group shares the latest updates of P2T and related tools:The author also maintainsPlanet of KnowledgeP2T/CnOCR/CnSTD Private Group, welcome to join. ThePlanet of Knowledge Private Groupwill release some P2T/CnOCR/CnSTD related private materials one after another, includingnon-public models,discount for paid models, answers to problems encountered during usage, etc. This group also releases the latest research materials related to VIE/OCR/STD.List of Supported LanguagesThe text recognition engine of Pix2Text supports80+languages, includingEnglish, Simplified Chinese, Traditional Chinese, Vietnamese, etc. Among these,EnglishandSimplified Chineserecognition utilize the open-source OCR toolCnOCR, while recognition for other languages employs the open-source OCR toolEasyOCR. Special thanks to the respective authors.List ofSupported LanguagesandLanguage Codesare shown below:↓↓↓ Click to show details ↓↓↓LanguageCode NameAbazaabqAdygheadyAfrikaansafAngikaangArabicarAssameseasAvaravaAzerbaijaniazBelarusianbeBulgarianbgBiharibhBhojpuribhoBengalibnBosnianbsSimplified Chinesech_simTraditional Chinesech_traChechencheCzechcsWelshcyDanishdaDargwadarGermandeEnglishenSpanishesEstonianetPersian (Farsi)faFrenchfrIrishgaGoan KonkanigomHindihiCroatianhrHungarianhuIndonesianidIngushinhIcelandicisItalianitJapanesejaKabardiankbdKannadaknKoreankoKurdishkuLatinlaLaklbeLezghianlezLithuanianltLatvianlvMagahimahMaithilimaiMaorimiMongolianmnMarathimrMalaymsMaltesemtNepalineNewarinewDutchnlNorwegiannoOccitanocPalipiPolishplPortugueseptRomanianroRussianruSerbian (cyrillic)rs_cyrillicSerbian (latin)rs_latinNagpurisckSlovakskSlovenianslAlbaniansqSwedishsvSwahiliswTamiltaTabassarantabTeluguteThaithTajiktjkTagalogtlTurkishtrUyghurugUkranianukUrduurUzbekuzVietnameseviRef:Supported Languages.UsageRecognizing Mixed Images with Both Text and FormulasFor mixed images containing both text and mathematical formulas, use the.recognize()function to identify the text and mathematical formulas in the image. For example, for the following image (docs/examples/en1.jpg):The method is as follows:frompix2textimportPix2Text,merge_line_textsimg_fp='./docs/examples/en1.jpg'p2t=Pix2Text()outs=p2t.recognize(img_fp,resized_shape=608)# You can also use `p2t(img_fp)` to get the same resultprint(outs)# If you only need the recognized texts and LaTeX representations, use the following line of code to merge all resultsonly_text=merge_line_texts(outs,auto_line_break=True)print(only_text)The returned resultoutsis adict, where the keypositionindicates Box location information,typeindicates the category information, andtextrepresents the recognition result. For more details, seeAPI Interfaces.Recognizing Pure Formula ImagesFor images containing only mathematical formulas, the function.recognize_formula()can be used to recognize the mathematical formula as a LaTeX expression. For example, for the following image (docs/examples/math-formula-42.png):The method is as follows:frompix2textimportPix2Textimg_fp='./docs/examples/math-formula-42.png'p2t=Pix2Text()outs=p2t.recognize_formula(img_fp)print(outs)The returned result is a string, which is the corresponding LaTeX expression. For more details, seeAPI Interfaces.Recognizing Pure Text ImagesFor images that contain only text and no mathematical formulas, the function.recognize_text()can be used to recognize the text in the image. In this case, Pix2Text acts as a general text OCR engine. For example, for the following image (docs/examples/general.jpg):The method is as follows:frompix2textimportPix2Textimg_fp='./docs/examples/general.jpg'p2t=Pix2Text()outs=p2t.recognize_text(img_fp)print(outs)The returned result is a string, which is the corresponding sequence of text. For more details, seeAPI Interfaces.ExamplesEnglishRecognition Results:Recognition Command:$p2tpredict-len-amfd-tyolov7--analyzer-model-fp~/.cnstd/1.2/analysis/mfd-yolov7-epoch224-20230613.pt--formula-ocr-config'{"model_name":"mfr-pro","model_backend":"onnx"}'--resized-shape768--save-analysis-resout_tmp.jpg--text-ocr-config'{"rec_model_name": "doc-densenet_lite_666-gru_large"}'--auto-line-break-idocs/examples/en1.jpgNote ⚠️: The above command uses premium models. A free version of the models can also be used as follows, although the results may be slightly inferior:$p2tpredict-len-amfd-tyolov7_tiny--resized-shape768--save-analysis-resout_tmp.jpg--auto-line-break-idocs/examples/en1.jpgSimplified ChineseRecognition Results:Recognition Command:$p2tpredict-len,ch_sim-amfd-tyolov7--analyzer-model-fp~/.cnstd/1.2/analysis/mfd-yolov7-epoch224-20230613.pt--formula-ocr-config'{"model_name":"mfr-pro","model_backend":"onnx"}'--resized-shape768--save-analysis-resout_tmp.jpg--text-ocr-config'{"rec_model_name": "doc-densenet_lite_666-gru_large"}'--auto-line-break-idocs/examples/mixed.jpgNote ⚠️: The above command uses premium models. A free version of the models can also be used as follows, although the results may be slightly inferior:$p2tpredict-len,ch_sim-amfd-tyolov7_tiny--resized-shape768--save-analysis-resout_tmp.jpg--auto-line-break-idocs/examples/mixed.jpgTraditional ChineseRecognition Results:Recognition Command:$p2tpredict-len,ch_tra-amfd-tyolov7--analyzer-model-fp~/.cnstd/1.2/analysis/mfd-yolov7-epoch224-20230613.pt--formula-ocr-config'{"model_name":"mfr-pro","model_backend":"onnx"}'--resized-shape768--save-analysis-resout_tmp.jpg-idocs/examples/ch_tra.jpgNote ⚠️: The above command uses premium models. A free version of the models can also be used as follows, although the results may be slightly inferior:$p2tpredict-len,ch_tra-amfd-tyolov7_tiny--resized-shape768--save-analysis-resout_tmp.jpg-idocs/examples/ch_tra.jpgVietnameseRecognition Results:Recognition Command:$p2tpredict-len,vi-amfd-tyolov7--analyzer-model-fp~/.cnstd/1.2/analysis/mfd-yolov7-epoch224-20230613.pt--formula-ocr-config'{"model_name":"mfr-pro","model_backend":"onnx"}'--resized-shape768--save-analysis-resout_tmp.jpg-idocs/examples/vietnamese.jpgNote ⚠️: The above command uses premium models. A free version of the models can also be used as follows, although the results may be slightly inferior:$p2tpredict-len,vi-amfd-tyolov7_tiny--resized-shape768--save-analysis-resout_tmp.jpg-idocs/examples/vietnamese.jpgModel DownloadFree Open-source ModelsAfter installing Pix2Text, the system willautomatically downloadthe model files and store them in~/.pix2text/1.0directory when you use Pix2Text for the first time (the default path under Windows isC:\Users\<username>\AppData\Roaming\pix2text\1.0).NoteIf you have successfully run the above example, the model has completed its automatic download and you can ignore the subsequent contents of this section.Paid ModelsIn addition to the above free open-source models, we also trained higher-accuracy formula detection and recognition models for P2T. They are used by theP2T Online Serviceon which you can try the performance. These models are not free (sorry open-source developers need coffee too🥤). SeePix2Text (P2T) | Breezedeus.comfor details.InstallWell, one line of command is enough if it goes well.pipinstallpix2textIf you need to recognize languages other thanEnglishandSimplified Chinese, please use the following command to install additional packages:pipinstallpix2text[multilingual]If the installation is slow, you can specify a domestic installation source, such as using the Aliyun source:pipinstallpix2text-ihttps://mirrors.aliyun.com/pypi/simpleIf it is your first time to useOpenCV, then probably  the installation will not be very easy.  Bless.Pix2Textmainly depends onCnOCR>=2.2.2, andtransformers>=4.37.0. If you encounter problems with the installation, you can also refer to their installation instruction documentations.WarningIf you have never installed thePyTorch,OpenCVpython packages before, you may encounter a lot of problems during the first installation, but they are usually common problems that can be solved by Baidu/Google.API InterfacesClass InitializerMain class calledPix2Text, with initialization function:classPix2Text(object):def__init__(self,*,languages:Union[str,Sequence[str]]=('en','ch_sim'),analyzer_config:Dict[str,Any]=None,text_config:Dict[str,Any]=None,formula_config:Dict[str,Any]=None,device:str=None,**kwargs,):The parameters are described as follows:languages(str or Sequence[str]): Sequence of language codes for text recognition; default is('en', 'ch_sim'), which means it can recognize English and Simplified Chinese;analyzer_config(dict): Configuration for the classifier model. Default toNonemeaning using default config (MFD Analyzer):{'model_name':'mfd'# can be 'mfd' or 'layout'}text_config(dict): Configuration for the general recognizer. Default toNonemeaning using default:{}formula_config(dict): Configuration for the formula recognizer. Default toNonemeaning using default:{}device(str): Specifies the computing resource to be used. Supports options like['cpu', 'cuda', 'gpu', 'mps']; the default isNone, which indicates automatic selection of the device.**kwargs(): Other reserved parameters. Currently not used.Class Function for RecognitionRecognizing Mixed Images containing both Text and FormulasThe text or Latex recognition of one specified image is done by invoking the class function.recognize()of classPix2Text. The class function.recognize()is described as follows.defrecognize(self,img:Union[str,Path,Image.Image],**kwargs)->List[Dict[str,Any]]:where the input parameters are described as follows.img(strorImage.Image): the path of the image to be recognized, or the imageImagethat has been read by usingImage.open().**kwargs: Can contain:resized_shape: Resize image width to this before processing. Default:700.save_analysis_res: Save analysis visualization to this file/dir. Default:Nonemeaning not saving.embed_sep: LaTeX delimiter for embedded formulas. Only useful with MFD. Default:(' $', '$ ').isolated_sep: LaTeX delimiter for isolated formulas. Only useful with MFD. Default:('$$\n', '\n$$').det_bbox_max_expand_ratio (float): Expand the height of the detected text bounding box (bbox). This value represents the maximum expansion ratio above and below relative to the original bbox height; default value is0.2.mfr_batch_size(int): The batch size used for MFR (Mathematical Formula Recognition) prediction; the default value is1.It returns alistofdict, eachdictcontains:type: The category of the recognized image;ForMFD Analyzer(Mathematical Formula Detection), the values can betext(pure text),isolated(mathematical formulas in isolated lines), orembedding(mathematical formulas embedded in lines).ForLayout Analyzer(Layout Analysis), the values correspond to the categories of layout analysis results.text: Recognized text or latex.position: Detected box coordinates,np.ndarray, with shape[4, 2].line_number: Exists only when usingMFD Analyzer. Indicates the line number (starting from 0) of the box. Boxes with the sameline_numberare on the same line.Note: This is new since P2Tv0.2.3. Not in previous versions.ThePix2Textclass also implements the__call__()function, which does exactly the same thing as the.recognize()function.  So you can call it like:frompix2textimportPix2Text,merge_line_textsimg_fp='./docs/examples/formula.jpg'p2t=Pix2Text(analyzer_config=dict(model_name='mfd'))outs=p2t.recognize(img_fp,resized_shape=608)# Equal to p2t(img_fp, resized_shape=608)print(outs)# To get just the text contents, use:only_text=merge_line_texts(outs,auto_line_break=True)print(only_text)Recognizing Pure Text ImagesThe class method.recognize_text()of the classPix2Textis used to perform text recognition on specified images. In this case, Pix2Text provides general text recognition functionality. The class function.recognize_text()is described as follows:defrecognize_text(self,imgs:Union[str,Path,Image.Image,List[str],List[Path],List[Image.Image]],**kwargs,)->Union[str,List[str]]:The input parameters are explained as follows:imgs(Union[str, Path, Image.Image, List[str], List[Path], List[Image.Image]]): The path of the image(s) to be recognized, orImageobjects that has been read in usingImage.open(). Supports a single image or a list of multiple images.kwargs: Other parameters passed to the text recognition interface.The return result is the recognized text string (when the input is multiple images, a list of the same length is returned).Recognizing Pure Formula ImagesThe class method.recognize_formula()of the classPix2Textis used to recognize mathematical formulas in specified images and convert them into Latex representation. The class function.recognize_formula()is described as follows:defrecognize_formula(self,imgs:Union[str,Path,Image.Image,List[str],List[Path],List[Image.Image]],batch_size:int=1,**kwargs,)->Union[str,List[str]]:The input parameters are explained as follows:imgs(Union[str, Path, Image.Image, List[str], List[Path], List[Image.Image]]): The path of the image(s) to be recognized, orImageobjects that has been read in usingImage.open(). Supports a single image or a list of multiple images.batch_size: The batch size for processing.kwargs: Additional parameters to be passed to the formula recognition interface.The return result is the recognized LaTeX representation string (when the input is multiple images, a list of the same length is returned).Script UsageP2Tincludes the following command-line tools.Recognizing a single image or all images in a directoryUse thep2t predictcommand to predict a single image or all images in a directory. Below is the usage guide:$p2tpredict-h
Usage:p2tpredict[OPTIONS]UsePix2Text(P2T)topredictthetextinformationinanimage

Options:-l,--languagesTEXTLanguageCodesforText-OCRtorecognize,separatedbycommas[default:en,ch_sim]-a,--analyzer-name[mfd|layout]WhichAnalyzertouse,eitherMFDorLayoutAnalysis[default:mfd]-t,--analyzer-typeTEXTWhichmodeltousefortheAnalyzer,'yolov7_tiny'or'yolov7'[default:yolov7_tiny]--analyzer-model-fpTEXTFilepathfortheAnalyzerdetectionmodel.Default:`None`,meaningusingthedefaultmodel--formula-ocr-configTEXTConfigurationinformationfortheLatex-OCRmathematicalformularecognitionmodel.Default:`None`,meaningusingthedefaultconfiguration--text-ocr-configTEXTConfigurationinformationforText-OCRrecognition,inJSONstringformat.Default:`None`,meaningusingthedefaultconfiguration-d,--deviceTEXTChoosetorunthecodeusing`cpu`,`gpu`,oraspecificGPUlike`cuda:0`[default:cpu]--image-type[mixed|formula|text]Whichimagetypetoprocess,either'mixed','formula'or'text'[default:mixed]--resized-shapeINTEGERResizetheimagewidthtothissizebeforeprocessing[default:608]-i,--img-file-or-dirTEXTFilepathoftheinputimageorthespecifieddirectory[required]--save-analysis-resTEXTSavetheanalysisresultstothisfileordirectory(If'--img-file-or-dir'isafile/directory,then'--save-analysis-res'shouldalsobeafile/directory).Setto`None`fornotsaving--rec-kwargsTEXTkwargsforcalling.recognize(),inJSONstringformat--auto-line-break/--no-auto-line-breakWhethertoautomaticallydeterminetomergeadjacentlineresultsintoasinglelineresult[default:no-auto-line-break]--log-levelTEXTLogLevel,suchas`INFO`,`DEBUG`[default:INFO]-h,--helpShowthismessageandexit.This command can be used toprint detection and recognition results for the specified image. For example, run:$p2tpredict-amfd--resized-shape608-idocs/examples/en1.jpg--save-analysis-resoutput-en1.jpgThe above command prints the recognition results, and it will also store the detection results in theoutput-en1.jpgfile, similar to the effect below:HTTP ServerPix2Textadds the FastAPI-based HTTP server. The server requires the installation of several additional packages, which can be installed using the following command.$pipinstallpix2text[serve]Once the installation is complete, the HTTP server can be started with the following command (-pfollowed by theport, which can be adjusted as needed).$p2tserve-len,ch_sim-amfdp2t servecommand usage guide:$p2tserve-h
Usage:p2tserve[OPTIONS]StarttheHTTPservice.

Options:-l,--languagesTEXTLanguageCodesforText-OCRtorecognize,separatedbycommas[default:en,ch_sim]-a,--analyzer-name[mfd|layout]WhichAnalyzertouse,eitherMFDorLayoutAnalysis[default:mfd]-t,--analyzer-typeTEXTWhichmodeltousefortheAnalyzer,'yolov7_tiny'or'yolov7'[default:yolov7_tiny]--analyzer-model-fpTEXTFilepathfortheAnalyzerdetectionmodel.Default:`None`,meaningusingthedefaultmodel--formula-ocr-configTEXTConfigurationinformationfortheLatexOCRmathematicalformularecognitionmodel.Default:`None`,meaningusingthedefaultconfiguration--text-ocr-configTEXTConfigurationinformationforText-OCRrecognition,inJSONstringformat.Default:`None`,meaningusingthedefaultconfiguration-d,--deviceTEXTChoosetorunthecodeusing`cpu`,`gpu`,oraspecificGPUlike`cuda:0`[default:cpu]-H,--hostTEXTserverhost[default:0.0.0.0]-p,--portINTEGERserverport[default:8503]--reloadwhethertoreloadtheserverwhenthecodeshavebeenchanged--log-levelTEXTLogLevel,suchas`INFO`,`DEBUG`[default:INFO]-h,--helpShowthismessageandexit.After the service starts, you can call the service in the following ways.PythonTo call the service, refer to the following method in the filescripts/try_service.py:importrequestsurl='http://0.0.0.0:8503/pix2text'image_fp='docs/examples/mixed.jpg'data={"image_type":"mixed",# "mixed": Mixed image; "formula": Pure formula image; "text": Pure text image"resized_shape":768,# Effective only when image_type=="mixed""embed_sep":" $,$ ",# Effective only when image_type=="mixed""isolated_sep":"$$\n,\n$$"# Effective only when image_type=="mixed"}files={"image":(image_fp,open(image_fp,'rb'))}r=requests.post(url,data=data,files=files)outs=r.json()['results']ifisinstance(outs,str):only_text=outselse:only_text='\n'.join([out['text']foroutinouts])print(f'{only_text=}')CurlUsecurlto call the service:$curl-Fimage=@docs/examples/mixed.jpg--form'image_type=mixed'--form'resized_shape=768'http://0.0.0.0:8503/pix2textOther LanguagePlease refer to thecurlformat for your own implementation.A cup of coffee for the authorIt is not easy to maintain and evolve the project, so if it is helpful to you, please consideroffering the author a cup of coffee 🥤.Official code base:https://github.com/breezedeus/pix2text. Please cite it properly.For more information on Pix2Text (P2T), visit:https://www.breezedeus.com/pix2text.