Package: neosca
NeoSCA简体中文|繁體中文|
EnglishNeoSCA is a fork ofXiaofei Lu'sL2 Syntactic Complexity Analyzer(L2SCA), with added support for Windows and an improved command-line interface for easier usage. NeoSCA is written by Tan, Long (谭龙)。It accepts written English texts and computes the following measures:the frequency of 9 structures in the text:words (W)sentences (S)verb phrases (VP)clauses (C)T-units (T)dependent clauses (DC)complex T-units (CT)coordinate phrases (CP)complex nominals (CN), and14 syntactic complexity indices of the text:mean length of sentence (MLS)mean length of T-unit (MLT)mean length of clause (MLC)clauses per sentence (C/S)verb phrases per T-unit (VP/T)clauses per T-unit (C/T)dependent clauses per clause (DC/C)dependent clauses per T-unit (DC/T)T-units per sentence (T/S)complex T-unit ratio (CT/T)coordinate phrases per T-unit (CP/T)coordinate phrases per clause (CP/C)complex nominals per T-unit (CN/T)complex nominals per clause (CP/C)ContentsHighlightsInstallUsageBasic UsageAdvanced UsageMiscCitingRelated EffortsLicenseContactHighlightsCross-platform compatibility: Windows, macOS, and LinuxFlexible command-line options to serve various needsSupports reading txt/docx/odt filesCustom syntactic structure search/calculationInstallInstall NeoSCATo install NeoSCA, you need to havePython3.7 or later installed on your system. You can check if you already have Python installed by running the following command in your terminal:python--versionIf Python is not installed, you can download and install it fromPython website. Once you have Python installed, you can installneoscausingpip:pipinstallneoscaIf you are in China and having trouble with slow download speeds or network issues, you can use the Tsinghua University PyPI mirror to installneosca:pipinstallneosca-ihttps://pypi.tuna.tsinghua.edu.cn/simpleInstall dependenciesNeoSCA depends onJava,Stanford Parser, andStanford Tregex. NeoSCA provides an option to install all of them:nsca--check-dependsCalled with the--check-depends,
NeoSCA will download and unzip archive files of these three to%AppData%(for Windows users, usuallyC:\\Users\\<username>\\AppData\\Roaming) or~/.local/share(for macOS and Linux users),
and set the environment variableJAVA_HOME,STANFORD_PARSER_HOME, andSTANFORD_TREGEX_HOME.If you have previously installed any of the three, you need to manually set the according environment variable.UsageNeoSCA is a command-line tool. You can see the help message by runningnsca --helpin your terminal.Basic usageSingle inputTo analyze a single file, use the commandnscafollowed by the file path.nsca./samples/sample1.txt
nsca./samples/sample1.docxTables, figures, images, and other unrelated elements (except for headers and footers, which will be automatically ignored) should be manually removed before docx/odt files are analyzed.After running the above command, aresult.csvfile will be generated in the current directory. You can specify a different output filename using-o/--output-file.nsca./samples/sample1.txt-osample1.csv# frequency output: ./sample1.csvWhen analyzing a file whose name includes spaces, it is important to enclose the file path in single or double quotes. Assume you have asample 1.txtto analyze:nsca"./samples/sample 1.txt"This ensures that the entire filename including the spaces, is interpreted as a single argument. Without the double quotes, the command would interpret "./samples/sample" and "1.txt" as two separate arguments and the analysis would fail.Multiple inputSpecify the input directory afternsca.nsca samples/              # analyze every txt/docx file under the "samples/" directory
nsca samples/ --ftype txt  # analyze only txt files under "samples/"
nsca samples/ --ftype docx # analyze only docx files under "samples/"Or simply list each file:cd./samples/
nscasample1.txtsample2.txtYou can also usewildcardsto select multiple files at once.cd./samples/
nscasample*.txt# every file whose name starts with "sample" and ends with ".txt"nscasample[1-9].txtsample10.txt# sample1.txt -- sample10.txtnscasample10[1-9].txtsample1[1-9][0-9].txtsample200.txt# sample101.txt -- sample200.txtAdvanced usageExpand wildcardsUse--expand-wildcardsto print all files that match your wildcard pattern. This can help you ensure that your pattern matches all desired files and excludes any unwanted ones. Note that files that do not exist on the computer will not be included in the output, even if they match the specified pattern.nscasample10[1-9].txtsample1[1-9][0-9].txtsample200.txt--expand-wildcardsTreat newlines as sentence breaksStanford Parser by default does not take newlines as sentence breaks during the sentence segmentation. To achieve this you can use:nscasample1.txt--newline-breakalwaysThe--newline-breakhas 3 legal values:never(default),always, andtwo.nevermeans to ignore newlines for the purpose of sentence splitting.
It is appropriate for continuous text with hard line breaks when just the non-whitespace characters should be used to determine sentence breaks.alwaysmeans to treat a newline as a sentence break, but there still may be
more than one sentences per line.twomeans to take two or more consecutive newlines as a sentence break.
It is for text with hard line breaks and a blank line between paragraphs.Configuration fileYou can use a configuration file to define custom syntactic structures to search or calculate.The default configuration filename for neosca isnsca.json, neosca will try to findnsca.jsonin current working directory. Alternatively, you can provide your own configuration file withnsca --config <your_config_file>. The configuration file should be in JSON format and named with.jsonextension.{"structures":[{"name":"VP1","description":"regular verb phrases","tregex_pattern":"VP > S|SINV|SQ"},{"name":"VP2","description":"verb phrases in inverted yes/no questions or in wh-questions","tregex_pattern":"MD|VBZ|VBP|VBD > (SQ !< VP)"},{"name":"VP","description":"verb phrases","value_source":"VP1 + VP2"}]}Above is a part of neosca's built-in structure definitions. Each definition follows a key-value pair format, where both the key and value should be enclosed in quotation marks.There are two approaches to define a structure: usingtregex_patternorvalue_source.tregex_patternrepresents the formal definition in Tregex syntax. Structures defined throughtregex_patternwill be searched and counted by running Stanford Tregex against input text. For instructions about how to write a Tregex pattern, see:Chapter 6 of Xiaofei's bookComputational Methods for Corpus Annotation and AnalysisPowerpoint tutorialfor Tregex by Galen AndrewThe TregexPattern javadoc pagevalue_sourcespecifies an arithmetic operation on values of other structures to calculate the value of the structure being defined.value_sourcecan include names of other structures, integers, decimals,+,-,*,/,(and).value_sourceare tokenized using Python's standard librarytokenize, which is specifically designed for Python source code. The name of a structure that is refered to in avalue_sourceshould adhere to the naming convention of Python variables (composed ofletters,numbers, andunderscores, cannot start with anumber;lettersrefer to those characters defined in the Unicode character database as "Letter", such as English letters and Chinese characters), or otherwise the name will not be correctly recognized.Thevalue_sourcedefinition can be nested, which means that dependant structures in turn can also be defined throughvalue_sourceand rely on others, forming a tree-like relationship. But the terminal structures must be defined bytregex_patternto avoid circular definition.Structures can be defined using eithertregex_patternorvalue_source, but not both simultaneously. Thenameattribute will be used for--selectoption. Thedescriptionattribute is optional, you can omit it for convenience.Select a subset of measuresNeoSCA by default outputs values of all of the available measures. You can use--selectto only analyze measures that you are interested in. To see a full list of available measures, usensca --list.nsca--selectVPTDC/C--sample1.txtTo avoid the program taking input filenames as a selected measure and raising an error, use--to separate them from the measures. All arguments after--will be considered input filenames. Make sure to specify arguments except for input filenames at the left side of--.Combine subfilesUse-c/--combine-subfilesto add up frequencies of the 9 syntactic structures of subfiles and compute values of the 14 syntactic complexity indices for the imaginary parent file. You can use this option multiple times to combine different lists of subfiles respectively. The--should be used to separate input filenames from input subfile-names.nsca-csample1-sub1.txtsample1-sub2.txt
nsca-csample1-sub*.txt
nsca-csample1-sub*.txt-csample2-sub*.txt
nsca-csample1-sub*.txt-csample2-sub*.txt--sample[3-9].txtSkip long sentencesUse--max-lengthto only analyze sentences with lengths shorter than or equal to 100, for example.nscasample1.txt--max-length100When the--max-lengthis not specified, the program will try to analyze sentences of any lengths, but mayrun out of memorytrying to do so.Reserve intermediate resultsNeoSCA by default only saves frequency output. To reserve the parsed trees, use-por--reserve-parsed. To reserve matched subtrees, use-mor--reserve-matched.nscasamples/sample1.txt-p# frequency output: ./result.csv# parsed trees:     ./samples/sample1.parsednscasamples/sample1.txt-m# frequency output: ./result.csv# matched subtrees: ./result_matches/nscasamples/sample1.txt-p-m# frequency output: ./result.csv# parsed trees:     ./samples/sample1.parsed# matched subtrees: ./result_matches/MiscPass text through the command lineIf you want to analyze text that is passed directly through the command line, you can use--textfollowed by the text.nsca--text'The quick brown fox jumps over the lazy dog.'# frequency output: ./result.csvJSON outputYou can generate a JSON file by:nsca./samples/sample1.txt--output-formatjson# frequency output: ./result.jsonnsca./samples/sample1.txt-osample1.json# frequency output: ./sample1.jsonJust parse text and exitIf you only want to save the parsed trees and exit, you can use--no-query. This can be useful if you want to use the parsed trees for other purposes. When--no-queryis specified, the--reserve-parsedwill be automatically set.nscasamples/sample1.txt--no-query# parsed trees: samples/sample1.parsednsca--text'This is a test.'--no-query# parsed trees: ./cmdline_text.parsedParse trees as inputBy default, the program expects raw text as input that will be parsed before querying. If you already have parsed input files, use--no-parseto indicate that the program should skip the parsing step and proceed directly to querying. When this flag is set, the is_skip_querying and reserve_parsed are automatically set as False.nscasamples/sample1.parsed--no-parseList built-in measuresnsca --listW:words
S:sentences
VP:verbphrases
C:clauses
T:T-units
DC:dependentclauses
CT:complexT-units
CP:coordinatephrases
CN:complexnominals
MLS:meanlengthofsentence
MLT:meanlengthofT-unit
MLC:meanlengthofclause
C/S:clausespersentence
VP/T:verbphrasesperT-unit
C/T:clausesperT-unit
DC/C:dependentclausesperclause
DC/T:dependentclausesperT-unit
T/S:T-unitspersentence
CT/T:complexT-unitratio
CP/T:coordinatephrasesperT-unit
CP/C:coordinatephrasesperclause
CN/T:complexnominalsperT-unit
CN/C:complexnominalsperclauseTregex interfaceNeoSCA has a Tregex command line interfacensca-tregex, which behaves similar astregex.shfrom Tregex package, with additional support for Windows.Lexical complexity analysisNeoSCA provides annsca-lcacommand to do the lexical complexity analysis mirroring the functionality ofLCA (Lexical Complexity Analyzer). Below are the available measures:Measures of Lexical Density and SophisticationMeasureFormulaLexical DensityLexical Sophistication-ILexical Sophistication-IIVerb Sophistication-ICorrected Verb Sophistication-IVerb Sophistication-IIMeasures of Lexical VariationMeasureFormulaNumber of Different WordsNumber of Different Words (first 50 words)Number of Different Words (expected random 50)Number of Different Words (expected sequence 50)Type-Token RatioMean Segmental Type-Token Ratio (50)Corrected Type-Token RatioRoot Type-Token RatioBilogarithmic Type-Token RatioUber IndexLexical Word VariationVerb Variation-ISquared Verb Variation-ICorrected Verb Variation-IVerb Variation-IINoun VariationAdjective VariationAdverb VariationModifier Variationnsca-lcasample.txt# single input filensca-lcasamples/# multiple input filesCitingIf you use NeoSCA in your research, please cite as follows.BibTeX@misc{tan2022neosca,title={NeoSCA: A Fork of L2 Syntactic Complexity Analyzer, version 0.0.55},author={Long Tan},howpublished={\url{https://github.com/tanloong/neosca}},year={2022}}APA (7th edition)Tan, L. (2022).NeoSCA(version 0.0.55) [Computer software]. Github. https://github.com/tanloong/neoscaMLA (9th edition)Tan, Long.NeoSCA. version 0.0.55, GitHub, 2022, https://github.com/tanloong/neosca.Also, you need to cite Xiaofei's article describing L2SCA.BibTeX@article{xiaofei2010automatic,title={Automatic analysis of syntactic complexity in second language writing},author={Xiaofei Lu},journal={International journal of corpus linguistics},volume={15},number={4},pages={474--496},year={2010},publisher={John Benjamins Publishing Company},doi={10.1075/ijcl.15.4.02lu},}APA (7th edition)Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing.International Journal of Corpus Linguistics, 15(4), 474-496.MLA (9th edition)Lu, Xiaofei. "Automatic Analysis of Syntactic Complexity in Second Language Writing."International Journal of Corpus Linguistics, vol. 15, no. 4, John Benjamins Publishing Company, 2010, pp. 474-96.If you use the lexical complexity analyzing feature, please cite Xiaofei's article about LCA.BibTeX@article{xiaofei2012relationship,author={Xiaofei Lu},title={The Relationship of Lexical Richness to the Quality of ESL Learners' Oral Narratives},journal={The Modern Language Journal},volume={96},number={2},pages={190-208},doi={https://doi.org/10.1111/j.1540-4781.2011.01232\_1.x},year={2012}}APA (7th edition)Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners' oral narratives.The Modern Language Journal, 96(2), 190-208.MLA (9th edition)Lu, Xiaofei. "The Relationship of Lexical Richness to the Quality of ESL Learners' Oral Narratives."The Modern Language Journal, vol. 96, no. 2, Wiley-Blackwell, 2012, pp. 190-208.Related effortsL2SCA, the original implementation, written in Python, byXiaofei LuL2SCA online, byHaiyang AiTAASSC, written in Python, byKristopher KyleL2SCA written in R, byThomas Gaillat, Anas Knefati, and Antoine LafontaineFSCA(French Syntactic Complexity Analyzer), written in R, byNate VandeweerdLicenseDistributed under the terms of theGNU General Public License version 2or later.ContactYou can send bug reports, feature requests, or any questions via:GitHub Issuestanloong@foxmail.com