Package: usfm-grammar
USFM-GrammarThe python library that facilitatesParsing and validation of USFM files usingtree-sitter-usfm3Conversion of USFM files to other formats (USX, dict, list etc)Extraction of specific contents from USFM files like scripture alone(clean verses), notes (footnotes, cross-refs) etcBuilt on python 3.10Installationpip install usfm-grammarThis requires a C compiler. On Windows, Microsoft Visual C++ 14.0 or above is required.
It is recommended that you updatepip,setuptoolsandwheel.UsageBy importing library in Python codefrom usfm_grammar import USFMParser, Filter

# input_usfm_str = open("sample.usfm","r", encoding='utf8').read()
input_usfm_str = '''
\\id GEN
\\c 1
\\p
\\v 1 test verse
'''

my_parser = USFMParser(input_usfm_str)

errors = my_parser.errors
print(errors)To convert to USXfrom lxml import etree

usx_elem = my_parser.to_usx() # default filter=ALL
print(etree.tostring(usx_elem, encoding="unicode", pretty_print=True))To convert to Dict/USJoutput = my_parser.to_usj() # default all markers

# filters out specified markers from output
# output = my_parser.to_usj(exclude_markers=['s1','h', 'toc1','toc2','mt'])

# retains only specified contents from output
# output = my_parser.to_usj(include_markers=['id', 'c', 'v']) 

# use predefined marker groups instead of listing them one by one
# output = my_parser.to_usj(include_markers=Filter.BCV+Filter.TEXT)

# for a flattened JSON removing nesting brought in by paragraphs, lists, quotes, tables and character level markups
# output = my_parser.to_usj(exclude_markers=Filter.PARAGRAPHS+Filter.CHARACTERS)

# To NOT concatinate text extracted from different markers
# output = my_parser.to_usj(exclude_markers=Filter.PARAGRAPHS+Filter.CHARACTERS, combine_texts=False) 

print(output)To understand more about howexclude_markers,include_markers,combine_textsandFilterworks refer the section onfiltering on USJTo save as jsonimport json
dict_output = my_parser.to_usj()
with open("file_path.json", "w", encoding='utf-8') as fp:
	json.dump(dict_output, fp)To convert to List or table like formatlist_output = my_parser.to_list() 
#list_output = my_parser.to_list([Filter.SCRIPTURE_TEXT])

table_output = "\n".join(["\t".join(row) for row in list_output])
print(table_output)To round trip with USJfrom usfm_grammar import USFMParser, Filter

my_parser = USFMParser(input_usfm_str)
usj_obj = my_parser.to_usj()

my_parser2 = USFMParser(from_usj=usj_obj)
print(my_parser2.usfm):warning: There will be differences between first USFM and the generated one in 1. Spaces and lines 2. Default attributes will be given their names 3. Closing markers may be newly addedTo remove unwanted markers from USFMfrom usfm_grammar import USFMParser, Filter, USFMGenerator

my_parser = USFMParser(input_usfm_str)
usj_obj = my_parser.to_usj(include_markers=Filter.BCV+Filter.TEXT)

my_parser2 = USFMParser(from_usj=usj_obj)
print(my_parser2.usfm)USJ to USX or Tablefrom usfm_grammar import USFMParser, Filter

my_parser = USFMParser(input_usfm_str)
usj_obj = my_parser.to_usj()

my_parser2 = USFMParser(from_usj=usj_obj)
print(my_parser2.to_usx())
# print(my_parser2.to_list())USX to USFM, USJ or Tablefrom usfm_grammar import USFMParser, Filter
from lxml import etree

test_xml_file = "sample_usx.xml"
with open(test_xml_file, 'r', encoding='utf-8') as usx_file:
    usx_str = usx_file.read()
    usx_obj = etree.fromstring(usx_str)

    my_parser = USFMParser(from_usx=usx_obj)
    print(my_parser.usfm)
    # print(my_parser.to_usj())
    # print(my_parser.to_list())From CLIusage: usfm-grammar [-h] [--in_format {usfm,usj,usx}]
                    [--out_format {usj,table,syntax-tree,usx,markdown,usfm}]
                    [--include_markers {book_headers,titles,...}]
                    [--exclude_markers {book_headers,titles,...}]
                    [--csv_col_sep CSV_COL_SEP] [--csv_row_sep CSV_ROW_SEP]
                    [--ignore_errors] [--combine_text]
                    infile

Uses the tree-sitter-usfm grammar to parse and convert USFM to Syntax-tree,
JSON, CSV, USX etc.

positional arguments:
  infile                input usfm or usj file

options:
  -h, --help            show this help message and exit
  --in_format {usfm,usj}
                        input file format
  --out_format {usj,table,syntax-tree,usx,markdown,usfm}
                        output format
  --include_markers {book_headers,titles,comments,paragraphs,characters,notes,study_bible,bcv,text,ide,usfm,h,toc,toca,imt,is,ip,ipi,im,imi,ipq,imq,ipr,iq,ib,ili,iot,io,iex,imte,ie,mt,mte,cl,cd,ms,mr,s,sr,r,d,sp,sd,sts,rem,lit,restore,p,m,po,pr,cls,pmo,pm,pmc,pmr,pi,mi,nb,pc,ph,q,qr,qc,qa,qm,qd,lh,li,lf,lim,litl,tr,tc,th,tcr,thr,table,b,add,bk,dc,ior,iqt,k,litl,nd,ord,pn,png,qac,qs,qt,rq,sig,sls,tl,wj,em,bd,bdit,it,no,sc,sup,rb,pro,w,wh,wa,wg,lik,liv,jmp,f,fe,ef,efe,x,ex,fr,ft,fk,fq,fqa,fl,fw,fp,fv,fdc,xo,xop,xt,xta,xk,xq,xot,xnt,xdc,esb,cat,id,c,v,text-in-excluded-parent}
                        the list of of contents to be included
  --exclude_markers {book_headers,titles,comments,paragraphs,characters,notes,study_bible,bcv,text,ide,usfm,h,toc,toca,imt,is,ip,ipi,im,imi,ipq,imq,ipr,iq,ib,ili,iot,io,iex,imte,ie,mt,mte,cl,cd,ms,mr,s,sr,r,d,sp,sd,sts,rem,lit,restore,p,m,po,pr,cls,pmo,pm,pmc,pmr,pi,mi,nb,pc,ph,q,qr,qc,qa,qm,qd,lh,li,lf,lim,litl,tr,tc,th,tcr,thr,table,b,add,bk,dc,ior,iqt,k,litl,nd,ord,pn,png,qac,qs,qt,rq,sig,sls,tl,wj,em,bd,bdit,it,no,sc,sup,rb,pro,w,wh,wa,wg,lik,liv,jmp,f,fe,ef,efe,x,ex,fr,ft,fk,fq,fqa,fl,fw,fp,fv,fdc,xo,xop,xt,xta,xk,xq,xot,xnt,xdc,esb,cat,id,c,v,text-in-excluded-parent}
                        the list of of contents to be included
  --csv_col_sep CSV_COL_SEP
                        column separator or delimiter. Only useful with
                        format=table.
  --csv_row_sep CSV_ROW_SEP
                        row separator or delimiter. Only useful with
                        format=table.
  --ignore_errors       to get some output from successfully parsed portions
  --combine_text        to be used along with exclude_markers or
                        include_markers, to concatinate the consecutive text
                        snippets, from different components, or notExample>>> python3 -m usfm_grammar sample.usfm --out_format usx

>>> usfm-grammar sample.usfm

>>> usfm-grammar sample.usfm --out_format usx

>>> usfm-grammar sample.usfm --include_markers bcv --include_markers text --include_markers s

>>> usfm-grammar sample-usj.json --out_format usfmFiltering on USJThe filtering on USJ, the JSON output, is a feature incorporated to allow data extraction, markup cleaning etc. The argumentsexclude_markersandinclude_markersin the methodsUSFMParser.to_usj()makes this possible. Also theUSFMParser.to_list(), can accept these inputs and perform similar operations. There is CLI versions also for these arguments to replicate the filtering feature there.include_markersOptional input parameter toto_usj()andto_listin python library and also in CLI whenformat=jsonorformat=table. Defaults toNone.When proivded, only those markers listed will be included in the output.include_markersis applied before applyingexclude_markers.exclude_markersOptional input parameter toto_usj()andto_listin python library and also in CLI whenformat=jsonorformat=table. Defaults toNone. When proivded, all markers except those listed will be included in the output.combine_textsOptional input parameter toto_usj()andto_listin python library and also in CLI whenformat=jsonorformat=table. Defaults toTrue. After filtering out makers like paragraphs and characters, we are left with texts from within them, if 'text-in-excluded-parent' is also not excluded. These text snippets may come as separate components in the contents list. When this option isTrue, the consequetive text snippets will be concatinated together. The text concatination is done in a puctuation and space aware manner. If users need more control over the space handling or for any other reason, would prefer the texts snippets as different components in the output, this can be set toFalse.usfm_grammar.FilterThis Class provides a set of enums that would be useful in providing in theexclude_markersandinclude_markersinputs rather than users listing out individual markers. The class has following optionsBOOK_HEADERS : identification and introduction markers
  TITLES : section headings and associated markers
  COMMENTS : comment markers like \rem
  PARAGRAPHS : paragraph markers like \p, poetry markers, list table markers
  CHARACTERS : all character level markups like \em, \w, \wj etc and their nested versions with +
  NOTES : foot note, cross-reference and their content markers
  STUDY_BIBLE : \esb and `cat
  BCV : \id, \c and \v
  TEXT : 'text-in-excluded-parent'To inspect which are the markers in each of these options, it could be just printed out,print(Filter.TITLES). These could be used individually or concatinated to get the desired filtering of markers and data:output = my_parser.to_usj(include_markers=Filter.BCV)
output = my_parser.to_usj(include_markers=Filter.BCV+Filter.TEXT)
output = my_parser.to_usj(exclude_markers=Filter.PARAGRAPHS+Filter.CHARACTERS)Inner contents of excluded markersFor markers like\p\qetc, by excluding them, we only remove them from the heirachy and retain the inner contents like\v, text etc that would be coming inside it. But for certain other markers like\f,\x,\esbetc, if they are excluded their inner contents are also excluded. Following is the set of all markers, who inner contents are discarded if they are mentioned inexclude_markersor not included ininclude_markers.BOOK_HEADERS, TITLES, COMMENTS, NOTES, STUDY_BIBLE:warning: Generally, it is recommended to NOT use bothexclude_markersandinclude_markerstogether as it could lead to unexpected behavours and data loss. For instance ifinclude_makershas\fkandexclude_markershas\f, the output will not contain\fkas all inner contents of\fwill be discarded.