Package: sfm-utils
sfm_utilsis a collection of python utilities to quickly and easily
summarise content and identify inconsistencies in lexicography data
encoded using Standard Format Markers (SFM data files).
Primarily these utilities are intended to provide assistance when
cleaning SFM data before converting to another format
or importing into a tool such asSIL Fieldworks Language Explorer (FLEx).SFM files contain lexicographical data
structured using tags (backslash codes). For example:\lx déláme
\ps n
\gn petite calebasse
\ps v
\gn sorte de verre
\ge drinking bowl
\gr ɓi loondesfm_utilsscripts do not attribute meaning to the tags and are
therefore independent of the set of tags used in an SFM
data file. The intent ofsfm_utilsis to ensure that tags are
used consistently throughout the data file.Author: Gavin Falconer (gfalconer@expressivelogic.co.uk)Installationsfm_utilsis distributed as a python package, so can beinstalled via pip(or your package manager of choice).
Requirespython v3or above:> pip install sfm_utilsFuture SuggestionUse a hosted version of sfm_utils within anonline jupyter notebookSee, for example:https://jvns.ca/blog/2017/11/12/binder–an-awesome-tool-for-hosting-jupyter-notebooks/IntroductionUsesfm-snifferto quickly get an insight into the content of any
SFM file.sfm-snifferlists the tags used in the file, giving the number
of occurrences of each tag. It also deduces a type for each tag, and shows
the number of ‘exceptions’, where the tag value did not match the expected type.> sfm-sniffer --summary my_lexicon.sfm
\gn : gloss (national)     : occurrences=2480 : type=text            : exceptions=26
\lx : lexeme               : occurrences=2474 : type=word            : exceptions=7
\sn : sense number         : occurrences=2456 : type=enumeration     : exceptions=28
\ps : part of speech       : occurrences=2450 : type=enumeration     : exceptions=79
\ge : gloss (english)      : occurrences= 511 : type=optional word   : exceptions=12
\gr : gloss (regional)     : occurrences= 500 : type=optional phrase : exceptions=11
\glo: ???                  : occurrences= 354 : type=text            : exceptions=0Runningsfm-snifferin full mode gives line references to pinpoint
exceptions:> sfm-sniffer my_lexicon.sfm
glo: gloss (other)        : occurrences= 354: type=text   : exceptions=0
===================================
\lx : lexeme              : occurrences=2474: type=word
7 exceptions for \lx of type 'word':
line    1: \lx <no value>
line 2335: \lx eptsá - v. int. fatsa
line 2470: \lx ékséɓé, ésséɓá
line 2474: \lx ékslá, alá
line 2712: \lx fá wé...
line 4025: \lx icá  - v.int. ɗatsa
line 11051: \lx ŋá (v.int. ŋɛŋa)
====================================
\ps : part of speech      : occurrences=2451: type=enumeration
Example values:
adj,adj adv,adj num,adj poss,adj poss.,adj?,adv,adv inter,adv tm,...
79 exceptions for \ps of type 'enumeration':
line  855: \ps v. int
line 1875: \ps v. int.
line 1879: \ps <no value>
line 1947: \ps <no value>
...The results indicate
the consistency of usage (or otherwise) for each tag. See theexample walkthroughfor more details.Tag Type DeductionTag type deduction works by examining the set of values used for each
tag. If the majority of values conform to a known type then the tag
is deduced to be of that type. (The threshold applied to determine
an acceptable majority can be varied by selecing a ‘strictness’ option.)The types are checked in order, with more specific types being
checked first. Therefore a tag will be deduced to be of the most
specific type that can be applied to the set of values used for that tag.Tag types may be one of the following (ordered from most specific to
least specific):OrderTypeDescription1NULL typeTag never has a value.2numberNumeric value, e.g. 1, 2, 3. The tag must have a value.3optional numberNumeric value, or may be empty.4enumerationA single word or phrase drawn from a limited
set of possible values. A typical example could be \ps (part of speech)
accepting one of: noun, verb, adjective, adverb,… The tag must have a value.5optional enumerationAs above, or may be empty.6wordA single-word value. A word may include
non-alphanumeric characters, but must include at least one alphanumeric
character. It may not include any whitespace, period, comma or semicolon
within the value. A trailing period, comma or semicolon is acceptable.
The following are all valid words:ésséɓá,up!,abbrev..
The tag must have a value.7optional wordAs above, or may be empty.8phraseA single-phrase value. Likewordbut may
contain whitespace. May not contain a period, comma or semicolon except
as a trailing character.up and away!is a valid phrase.up; away!is not (it is assumed to be a list value).
The tag must have a value.9optional phraseAs above, or may be empty.10enumeration listA list of words or phrases (separated by
commas or semicolons) where each word or phrase is drawn from a
limited set of possible values. The tag must have a value.11textAny combination of characters, words or
phrases. The tag must have a value.12optional textAny combination of characters, words or
phrases, or may be empty. Theoptional texttype is generic, and
indicates that no consistent pattern of usage could be deduced for the
tag.Coming Soon…Usesfm-struct-snifferto analyse the tree structure of the SFM
file and generate a proposed schema:> sfm-struct-sniffer my_lexicon.sfm > my_lexicon.schemaThen usesfm-struct-snifferto verify the integrity of the SFM
data against the schema:> sfm-struct-sniffer --verify --schema=my_lexicon.schema my_lexicon.sfm
...The generated schema is a simple text file so can easily be modified:\lx
    \ps
        \ge
        \go?
        \sn?
            \ge
            \go?When it becomes necessary to edit or correct the SFM file by hand, the
data can be formatted bysfm-struct-snifferto apply indentation
that shows the tree structure:> sfm-struct-sniffer --format -schema=my_lexicon.schema my_lexicon.sfm
\lx déláme
    \ps n
        \gn petite calebasse
    \ps v
        \gn sorte de verre
        \ge drinking bowl
        \gr ɓi loonde
 \lx deremke
    \ps num
        \gn cent
        \ge one hundred
        \gr temerreThis also makes it easier to reason about the outcomes of importing
the data intoSIL Fieldworks Language Explorer (FLEx)Future Suggestionsfm-struct-sniffercould embed comments in the file to
highlight exceptions or ambiguous tree elements, e.g:\lx déláme
   \ps n
# >>> unexpected \sn
      \sn 1
# <<<FeaturesWorks with any SFM file. Inferred types are the result of statistical
analysis on the SFM file contents. No semantics are assumed, no
a priori knowledge is ncessary.UsageUsage information forsfm-sniffercan be shown by using the –help
option. See also theexample walkthrough.Usage:sfm-sniffer [--tags=<dictionary>] [--summary] [--normal|--stricter|--strictest] <file>
sfm-sniffer --dumptags
sfm-sniffer (-h | --help)
sfm-sniffer --versionOptions:-t,--tags=fileRead a dictionary file that maps tags to labels.
If unspecified, the defaultMDFtag labels will be used.[1]-s,--summaryOutput a summary report only.-1,--normalApply normal type deduction rules.-2,--stricterApply stricter type deduction rules.-3,--strictestApply strictest type deduction rules.-d,--dumptagsPrint the default SFM tag dictionary in the format
used by –tags-h,--helpShow this screen.--versionShow version.Applying stricter type deduction rules will generate a report that
prefers more specific types (such asnumberorword) over more
general types (such asoptional text). However, stricter type
deduction rules are more likely to generate a large number of exceptions.Similarily, forsfm-struct-sniffer:Usage:sfm-struct-sniffer [--tags=<dictionary>] <file>
sfm-struct-sniffer --dumptags
sfm-struct-sniffer (-h | --help)
sfm-struct-sniffer --versionOptions:-t,--tags=fileRead a dictionary file that maps tags to labels.
If unspecified, the defaultMDFtag labels will be used.[1]-d,--dumptagsPrint the default SFM tag dictionary in the format
used by –tags-h,--helpShow this screen.--versionShow version.Repository contentsTODOSee AlsoSOLIDis an existing graphical utility provided by SIL
to check, clean up, and convert SDF files.ReferencesMaking Dictionaries: A guide to lexicography and the Multi-Dictionary Formatter(Coward & Grimes, 2000): a description of theMDF(Multi-Dictionary Formatter) and the
defined set of SFM backslash codes that are commonly recognised.Technical Notes on SFM Database Import(Ken Zook, 2010):
provides further information on issues that are likely to be encountered
when working with SFM files.