Change log
==========

AUG 20
------
- Added comments to wordLevelPedagogAnnotations.pl
- Trying to optimize the process

AUG 19
------
- Added comments to countPedagogicalFeatures.pl and to wordLevelPedagogAnnotations.pl

AUG 7
-----
- Udpdated README.md, INSTALL, USAGE and EXTENDING
- Updated $SPINTX_HOME/docs/PAToS-DataFlowDiagram.odg

AUG 6
-----
- Tuned up rules to annotate describir personas (carácter)
- Added some rules to correct (post-edit) some of tree-tagger's decisions
- Updated PDF grammar documentation

AUG 5
-----
- Tuned up rules to handle dar and pedir opiniones in progress
- Started working annotating describir personas (carácter) but needs a lot of tunning up yet

AUG 2
-----
- New rules to handle dar and pedir opiniones in progress

JULY 31
-------
- Added rules to annotate pedir and ofrecer ayuda
- Added rules to annotate referir palabras (indirect speech)
- Updated PDF docu
- Added rule to repair dijo's reading (present indicative to pretérito perfecto)

JULY 30
-------
- Added some more rules to handle Func:Necesidades
- Added rules to handle Func:Agradecer

JULY 29
-------
- Added rules to handle Func:Necesidades
- Added some more rules for algo and nada
- Added some more rules to post-edit some of TreeTagger's decisions

JULY 23
-------
- New rules annotating code switching
- New rules annotating pronoun uses of nada and algo

JULY 22
-------
- Added more rules to handle CuantIndef
- Added rules to handel some cases of todo X, todo DET X
- Added repairing rules for y, o and a (related to ALFS)

JULY 22
-------
- Added VSfin and VMfin to the set FiniteVerbForms (which precluded certain annotations to take place)
- Added new rules to further indentify Gram:Det:CuantIndef
- Added rules to identify det combinations like "los demás", "los pocos" or "muchos otros" (the classification of "otro" as determiner is controverted, but currently we assume it (and leave a possible discussion for a later moment).
- Several updates in the grammar documentation

JULY 19
-------
- Continuation with cuantitativos and indefinidos
- Found a lot of inconsistent decisions on the side of TreeTagger regarding cuantitativos and indefinidos and decided to retag them to to other tags

JULY 18
-------
- Started to add new rules of the identification of cuantitativos e indefinidos

JULY 17
-------
- Added/Extended rules to handle lo + adj or lo + que in a more fine-grained fashion
- Added rules to handle in a unified manner things like quizá(s), tal vez and a lo mejor
- Repaired bug in rule about future tenses
- Update grammar documentation in PDF doc.

JULY 15
-------
- Added rule to handle Buenos Aires as a compound name (not as two different words)
- Added rules to handle recomendaciones based on es importante/mejor que...
- Changed ProcessSPinTX.sh: it had a bug that deleted cg3 files when called using cg3 or cg3trace (instead of deleting *.out files)
- Update grammar documentation in PDF doc.

JULY 11
-------
- Minor changes in scripts to adapt to column names in Google Fusion tables
- Changes in documentation generation process to add acknowledgements
- Updated pdf documentation of the annotation grammar

JULY 9, 2013
------------
- Added some comments to ProcessSPinTX.sh for documentation purposes
- Added a patch to add gender Fem or Masc to definite articles el, la, los, las so that CQP does not complain for compilation (this is not of much use to pedagogical annotation right now, but it is for practical purposes when using CQP to inform rule writing)
- Added code to remove older files in folders that should only contain files that result from processing the files in ClipTags (affects cg3, ClipPedagogicalTags and ClipWLA)
- Added patch to ttagger2OtherFormats.pl to avoid annotationg _AllClips.txt
- Starting to work on annotation of future (conjugado y de perífrasis)

JULY 8, 2013
------------
- Added option cg3trace to PorcessSPinTX.sh so that cg3 can be executed with trace option easily by grammar writers.
- Added .gitignore files to some of the folders

JUNE 27, 2013 (bis)
-------------
- Added new PDF documentation of the pedagogical annotation grammar.

JUNE 27, 2013
-----------
- Changed countPedagogicalFeatures.pl to allow for a Func column in ClipPedagogicalMetadata.tsv (as said before this is currently not clean enough and has to be done manually)
- Changed rules that established relations between words occuring further left to words occuring further right. This created weird higlihthings on the web, for instance, in sentence like "Dime, le gusta vivir en Estados Unidosa tu hijo." where the expression "le gusta" is marked as relevant for some grammar points, "le" was marked as the end point and "gusta" as the start point and that would make things weird in displaying.
- Also changed ProcessSPinTX.sh to adapt to the Func annotations. Otherwise the ClipMetadataPedagogical.tsv table was not properly generated.

JUNE 27, 2013
-----------
- Added rules to repair readings for para and creen.
- Added more rules to handle exponents of function for deseos and purposes using para.

JUNE 25, 2013
-----------
- Adapted rule to handle desire expressions using nouns like necesidad de que VERB+MSubj so that there can be a NP or a clitic pronoun between the que and the verb (using the function TEMPLATE of CG3).

JUNE 25, 2013
-----------
- Adding some more rules for handling deseos y recomendaciones.

JUNE 19, 2013
-----------
- Changes in GrammarDocuSPinTX.pl so that it works with the appropriate paths (after slight changes in path names)
- Changes in GrammarDocuSPinTX.pl so that it works out rules that have low long dashes in them like _ (which caused problems in pdflatex compilation for the generation of the PDF documentation of the rules)

JUNE 18, 2013
-----------
- Added rules to handle deseos, recomendaciones and a couple more of functional tags (still underway)

JUNE 17, 2013
-----------
- Rules to handle esto, eso, aquello as pronouns (in fact marked as determiners with a pronominal function, following certain theories that advocate for the inexistence of empty categories).
- Rules to handle "uno" as an indefinite article in as in "uno de sus hermanos".
- Halfway working on the indentification of exponents of function for "expresar deseos y recomendaciones"

JUNE 13, 2013
-----------
- Added rules to handle mandatos informales en singular (tú)

JUNE 12, 2013
-----------
- Added rules to disambiguate dar|decir occurrences of "di" into the dar (to better handle imperativo)
- Changed internal path definitions to work properly with installation in virtual server (basically assumes that there is a SPINTX_HOME variable that has two subfolders data and tools. These two folders contain the rest of files (downloaded mainly from github or processed internally if there is the need)
- Changed the way the CG3 grammar was called by ProcessSPinTX.sh so that it adapts (hopefully seamlessly) to new installations

JUNE 10, 2013
-----------
- Added rules to handle ordinal adjectives (a subset at least)
- Added rules to modify reading "UNK" NC Plural of mis and set it to "mío" PPO Plural (consistent with "mi")
- Updated docu
- Update excel and fusion tables accordingly (in Google Drive)

MAY 28, 2013
-----------
- Rules to handle annotation of masc nouns that end in -a (problema, tema, día)
- Rules to annotate fem nouns that require el as article (el alma, el área)
- Added corresponding documentation

MAY 28, 2013
-----------
- Added option to make pdflatex compilation optional in GrammarDocuSPinTX.pl
- Added/Refined rules to handle haber as auxiliar
- Excluded and justified why ser cannot be tagged as auxiliar (right now) -- at least it does not seem worth the effort
- New version of grammard documentation in PDF (see docs folder)
- Repaired bug in formatting of documentation of rule related information
- Repaired bug in annotation of tu as subject pronoun (when it has no tilde it is usually a determiner, unless it is a typo)
- Rules for the annotation of possessive determiners
JUNE 10, 2013
-----------
- Added rules to handle ordinal adjectives (a subset at least)
- Added rules to modify reading "UNK" NC Plural of mis and set it to "mío" PPO Plural (consistent with "mi")
- Updated docu
- Update excel and fusion tables accordingly (in Google Drive)

MAY 28, 2013
-----------
- Rules to handle annotation of masc nouns that end in -a (problema, tema, día)
- Rules to annotate fem nouns that require el as article (el alma, el área)
- Added corresponding documentation

MAY 28, 2013
-----------
- Added option to make pdflatex compilation optional in GrammarDocuSPinTX.pl
- Added/Refined rules to handle haber as auxiliar
- Excluded and justified why ser cannot be tagged as auxiliar (right now) -- at least it does not seem worth the effort
- New version of grammard documentation in PDF (see docs folder)
- Repaired bug in formatting of documentation of rule related information
- Repaired bug in annotation of tu as subject pronoun (when it has no tilde it is usually a determiner, unless it is a typo)
- Rules for the annotation of possessive determiners

MAY 25, 2013
-----------
- Created rules to correct ser reading for fueron in several contexts (not "ser")
- Created rules to correct wrong adj reading for maestro/a/os/as
- Created rules to disambiguate erar|ser readings for eran and eras, leave them as ser
- Added label for haber as auxiliary so that "has visto" can be accessed as a form of presente perfecto but also as a form of haber used as an auxiliary verb
- Added some rules to start annotating "ha sido" as a non-auxiliary use of ser if not followed by a participle

MAY 24, 2013
-----------
- Added script CompileSPinTXInCQP.sh, which compiles the linguistically (not pedagogically) annotated files to CWB assuming this is installed and you know how to work with it
- Modified ttagger2OtherFormats.pl so that it can generate tsv files in ClipCQP folder to work with Corpus WorkBench format
- Updated README.md with instructions on how to use CompileSPinTXInCQP.sh

MAY 23, 2013
-----------
- Adapted scripts to work with github's folder structure
- Changed files with extension solr to extesion pla (stands for Pedagogical List of Annotations). These files are stored in ClipPedagogicalTags (folder in the github repository https://github.com/coerll/SpinTXCorpusData)

MAY 17, 2013
-----------
- Updated CG grammar documentation
- Updated last few changes, mainly minor changes in CG grammar, but forgot which (sorry).
- Changed typology classes and names for the annotation of haber as an existential verb as in 'Hay tres cervezas en la nevera'.

APR 25, 2013
-----------
- Now n-gram vocab annotations appear in ClipLevelMetadata.tsv

APR 23, 2013
-----------
- Removed from the one word vocab list those words that belong to two-word vocab items in another list. For instance, El Paso appears as Vocab item, but Paso not anymore.

APR 9, 2013
-----------
- Added a patch to generate annotation record lines for 1st and 2nd level of pedagogical annotation info

  What used to be (in *.wla.json files and *.tsv files):

AF002_1989_EP_SU2011_AD_01      Prep    Gram    tt-29   tt-29
AF002_1989_EP_SU2011_AD_01      RecuDiscurso:GanarTiempo        Prag    tt-32   tt-32

  Now is:

AF002_1989_EP_SU2011_AD_01      Prep    Gram    tt-29   tt-29
AF002_1989_EP_SU2011_AD_01      RecuDiscurso    Prag    tt-32   tt-32
AF002_1989_EP_SU2011_AD_01      RecuDiscurso:GanarTiempo        Prag    tt-32   tt-32
- Started writing a new script to generate a CQP formatted file out of the *.out files to allow better searching possibilities to grammar developers (and as a by product for research in linguistics), cg32cqp.pl
- Adapted rules to merge different pedagogical labels used for tagging of constructions with object pronouns to one single label (Gram:Pron:Objeto)

APR 4, 2013
-----------
- Changed rule code @Gram:PretPerf to @Gram:Indefinido (as it stands in the typology doc)
- Changed all rules with a code such as @:Gram:ExpresarSorpresas:* to Gram:Gustar:Otros (to be coherent with current typology)
- Changed code for "tal vez" from Prag:MarcadoresDisc:NoTipificado to Prag:MarcadoresDisc:Igualdad, I think this is the most coherent classification following Laura Alonso's criteria (whose work I used to classify our discourse markers)
- Added rules to annotate present and past subjunctive and the plu- versions (hayas cantado and hubieras cantado)

MAR 28, 2013
------------
- Added comma after start info on json format for word level annotation (ClipsWLA/*.wla.json)
- Changed internal and output file names to be coherent with table names in Google Drive (SPinTX LIFT > Clips Corpus > ClipsDataTables)
- Added rules to handle bigrams corresponding to geographical entities (El Paso, Estados Unidos, Ciudad Juárez)
- Added rules to handle bigrams corresponding to verb-preposition combinations (pensar, hablar, empezar)

MAR 26, 2013
------------
- put again the comma in the json format files for word level information between different token info lines

MAR 21, 2013
------------
- Changed output file names and extensions to match spintx-wide ones
- Changed format of Vocab metadata, because it had an error (tag and tag_type where not in different columns)

MAR 19, 2013
------------
- Clip level summary of vocab info implemented
- Generation one corpus word level annotation table in CSV format
- Updated DataFlowDiagram.odg in docs folder
- Updated the USAGE file in the main repo folder
- Changed wla folder name to ClipsWLA (so that it is compatible with the Google Drive naming conventions)

MAR 8, 2013 (bis)
-----------------
- Half way of implementing the clip level summary of vocabulary info in wordLevelPedagogAnnotations.pl.

MAR 8, 2013
-----------
- Improved ProcessSpintx.sh help page and implmented individual execution options for each step in annotation process.
- Repaired bug in wordLevelPedagogAnnotations.pl that incrementally increased size of VocabList array.

MAR 5, 2013
-----------
- Adapted ttagger2OtherFormats.pl to process input files where punctuation is no longer a token property but a token in itself.

FEB 26, 2013
------------
- Added some internal documentation on the code to wordLevelPedagogAnnotations.pl
- Started to generate uni-gram counts for the generation of vocabulary lists

FEB 21, 2013 (PM)
-----------------
- Changed format of reading info in CG3 compatible cohorst for language. Now it is represented as "lang:" and the ccode of the language in which the word is (es for Spanish, and en for English).
- Added a sample rule in CG3 grammar to exemplify how the annotation of code switching could be implemented. It is far from being solved, it is just a proof of concept.
- Added rules for Gram:Det:Demo, Prag:RecDocu:GanarTiempo.
- Upddated PDF doc for rule specificaitons in docs folder under $SPINTX_HOME/spintxPedagogicalAnntotation.
- Changed countPedagogicFeatures.pl to allow for the counting of annotations of the main category Herit (this is kind of ugly, but currently new 1st level classes have to be encoded manually in this one script).

FEB 21, 2013 (AM)
-----------------
- Changed format of JSON files to [{...},{...},{...}].
- wla.json files now dumped to wla folder in $SPINTX_HOME/corpus/ClipTags/wla and a copy of ClipsWLAOneFile.json is created in $SPINTX_HOME/corpus/ClipTags/stats folder for the records including time and date.
- Enhanced and added info on INSTALL, USAGE and EXTENDING file.

FEB 19, 2013
------------
- Modified GrammarDocuSPinTX.pl to ensure that it generates the *.tex file and *.pdf file in $SPINTX_HOME/spintxPedagogicalAnntotation/docs.
- Added INSTALL file, including requirements.
- Added USAGE file, with basic instructions on how to execute it.

FEB 14, 2013
------------
- Changed JSON formatted word level metadata to use labels clip, start, end and separate pedagogical lable into type (Gram, Prag, etc.) or tag (Aux:Estar...)
- Added data flow diagram to docs/PATos-DataFlowDiagram.odg
- Modified script ProcessSPinTX.sh so that it runs all the required scripts from import of lemmatised and POS-tagged files to returning the clip-level information and word-level annotation files.

FEB 12, 2013
------------
- Now generating output of word-level information in JSON format.

FEB 7, 2013
-----------

- Commented pedagogical tags with four levels of description. Now three is the deepest level a pedaogical annotation should show.
- Added annotation rules for "se", "ser", "estar" and "haber".
- Changed output format of clip level information breakdown to ClipId TAB GramTag1,GramTag2 TAB PragTag1, PragTag2...