Package: camel-morph
Camel MorphThe work presented in this repository is part of a large effort on Arabic morphology under the name of the Camel Morph Project [^1] developed by theCAMeL LabatNew York University Abu Dhabi.Please useGitHub Issuesto report a bug or if you need help using Camel Morph.Camel Morph’s goal is to build large open-source morphological models for Arabic and its dialects across many genres and domains. This repository contains code meant to build an ALMOR-style database (DB) from a set of morphological specification and lexicon spreadsheets, which can then be used byCamel Toolsfor morphological analysis, generation, and reinflection.The following sections provide useful usage information about the repository. For pointers to the system used for the SIGMORPHON 2022 Camel Morph paper, check theCamel Morph SIGMORPHON 2022section.Camel Morph SIGMORPHON 2022For instructions related to inspecting, making use of, replicating the results obtained for the SIGMORPHON 2022 Camel Morph paper[^1], and the data, see theCamel Morph SIGMORPHON 2022section in the./sigmorphon2022_releasefolder.DataThe data throughout this project is being maintained through the Google Sheets interface which can be used to add, delete, or edit morphological specification entries. The following are links to the data and morphological specifications used for this project, and areonly accessible upon demand.Data LicenseThe data files accessed through the below links are licensed under aCreative Commons Attribution 4.0 International License. For code license, seeLicense.Continuously Updated DataMSA Verbs SpecificationsEGY Verbs SpecificationsSIGMORPHON 2022 Data (frozen)The following data is not accessile publicly from the Google Sheets interface but is available incsvformat (the way it was at submission time) in the followingfolder.MSA Verbs Specifications (Camera Ready)EGY Verbs Specifications (Camera Ready)Folder HierarchyBelow, you will find a table describing the function of each directory contained in the repository.DirectoryDescription./camel_morphDirectory (in package format) containing all the necessary files to build, debug, test, and evaluate the Camel Morph DB Maker../configsContains configuration files which make running the scripts in the above directory easier../dataContains, for each different configuration, the set of morphological specification files necessary to run the different scripts../databasesContains the output DB files resulting from the DB Making process../debugging_outputContains all DB debugging files which are usually uploaded (automatically) to and analyzed in Google Sheets../eval_filesContains files necessary to carry out evalutation of generated DBs and analyzable evaluation output files (generally analyzed on Google Sheets)../misc_filesContains miscellaneous files used by scripts inside./camel_morph../sandboxContains various standalone files that are not used in./camel_morph../sigmorphon2022_releaseStandalone environment allowing users to run the DB Maker and Camel Tools engines without installing Camel Tools, in the same version used for the SIGMORPHON 2022 paper. Also contains the data that was used to get the results described in the paper[^1].InstructionsTo generate databases, paradigm-specific inflection (conjugation/declension) tables, or evaluation tables, follow the below instructions. Note that the adjoiningMakefilecan be used to reproduce the experiments which were ran in-house by providing the (pipelines of) instructions that were used to carry them out. But first, it is important to understand the structure of the configuration file. Also, a default configuration file is included in the./configsdirectory for direct usage.InstallationTo start working with the Camel Morph environment:Clone (download) this repository and unzip in a directory of your choice.Make sure that you are runningPython 3.8orPython 3.9.Run the following command to install all needed libraries:pip install -r requirements.txt.Run all commands/scripts from the outercamel_morphdirectory.For development purposes only:Clone (download) aforkof the Camel Tools repository. The Camel Morph databases will currently only function using the latter instance of Camel Tools. The changes in this fork will eventually be integrated to the main Camel Tools library. Unzip in a directory of your choice.Set theCAMEL_TOOLS_PATHvalue to the path of the Camel Tools fork repository in the configuration file that you will be using (default configuration file./configs/config_default.jsonprovided; seeConfiguration File Structuresection).For instructions on how to run the different scripts, see the below sections.Compiling a Database (DB Maker)The below command compiles an ALMOR-style database starting from a set of morphological specification files referenced in the specific configuration mentioned as an argument.Usageusage:db_maker.py[-h][-config_fileCONFIG_FILE][-config_nameCONFIG_NAME][-output_dirOUTPUT_DIR][-run_profiling][-camel_tools{local,official}]Argumentsshortdefaulthelp-hShow this help message and exit.-config_fileconfig_default.jsonPath of the configuration file which contains different configurations to run the DB on. Some pre-compiled configurations already exist in./configs/config.json, but new ones could be easily added. Seeherefor an overview of the configuration file format. Defaults to./configs/config_default.json.-config_namedefault_configConfiguration name of one of the configurations contained inCONFIG_FILE. It contains script parameters, sheet paths, etc.-output_dirOverrides path of the directory to output the DBs to (specified in the global section ofCONFIG_FILE).-run_profilingTo generate an execution time profile of the specific configuration.-camel_toolslocalPath of directory containing the CAMeL Tools modules (should be cloned as describedhere).Configuration File StructureIn its most basic format, the configuration file should look like the example below in order to successfully run the scripts described in this guide. Unless otherwise stated, variables (beginning with$) are double quoted strings.Default Configuration File{
    "global": {
        "data_dir": $DATA_DIR_PATH,
        "specs": {
            "sheets": [
                $ABOUT_SHEET,
                $HEADER_SHEET
            ]
        },
        "db_dir": $DB_OUTPUT_DIR,
        "camel_tools": $CAMEL_TOOLS_PATH
    },
    "local": {
        $CONFIG_NAME: {
            "dialect": $DIALECT,
            "cat2id": $CAT2ID,
            "pruning": $PRUNING,
            "specs": {
                "sheets": {
                    "order": $ORDER_SHEET,
                    "morph": $MORPH_SHEET
                }
            },
            "lexicon": {
                "sheets": [
                    $LEX_SHEET_1,
                    ...
                ]
            },
            "db": $DB_NAME,
            "pos_type": $POS_TYPE
        }
    }
}where:$DATA_DIR_PATH: path of the outermost data directory where all sheets are kept (e.g.,data; referenced from the main repository directory). Sheets for this configuration should be kept inside a folder which has the name of the configuration ($CONFIG_NAME) which is itself contained in a directory calledcamel-morph-$DIALECT(where$DIALECTis specified below). So for example, if$DATA_DIR_PATH=data,$DIALECT=msa, and$CONFIG_NAME=pv_msa, then sheets for this configuration should be in a directory with the path./data/camel-morph-msa/pv_msa.$ABOUT_SHEET: name of the sheet containing theAboutsection which will go in the DB (e.g.,About). Downloaded as specified in theGoogle Sheetssection.$HEADER_SHEET: same as$ABOUT_SHEET(e.g.,Header)$DB_OUTPUT_DIR: name of the directory to which the compiled DBs will be output.$CAMEL_TOOLS_PATH: path of the Camel Tools repository fork that should be cloned/downloaded as described inInstallationsection.$CONFIG_NAME: name of the configuration in thelocalsection of the config file, to choose between a number of different configurations (e.g.,default_config). This is also the name of the folder which contains the sheets that are specified for that configuration and the global section.$DIALECT: dialect being worked with (i.e.,msaoregy). This is specified to further organize the configuration-specific data into high-level projects (i.e.,./data/camel-morph-msaor./data/camel-morph-egy).$CAT2ID: boolean (trueorfalse). Specifies the format in which to output the category names. If set to true, then category names are IDs, otherwise, they contain condition information.$PRUNING: boolean (trueorfalse). Used in the DB making process to speed up DB compilation. For this to be set totrue, the Morph sheet must contain condition definitions (organization of conditions into categories).$ORDER_SHEET: same as$ABOUT_SHEET(e.g.,MSA-Verb-ORDER).$MORPH_SHEET: same as$ABOUT_SHEET(e.g.,MSA-Verb-MORPH).$LEX_SHEET_1: same as$ABOUT_SHEET(e.g.,MSA-Verb-LEX-PV). At least one lexicon sheet can be specified; the latter will be concatenated in pre-processing.$DB_NAME: name of the output DB.$POS_TYPE: type of the POS for which we are building the DB. Can either beverbalornominal.Code LicenseAll the code contained in this repository is available under the MIT license. See theLICENSEfile for more info.ContributorsChristian Khairallah (Cayralat)Salam KhalifaNizar Habash[^1]: Nizar Habash, Reham Marzouk, Christian Khairallah, and Salam Khalifa. 2022.Morphotactic Modeling in an Open-source Multi-dialectal Arabic Morphological Analyzer and Generator. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 92–102, Seattle, Washington. Association for Computational Linguistics.
[^2]: Note that for the release directory, only the morphological components from Camel Tools were sourced from the actual library and were added to be imported locally.