Package: openedx-caliper-tracking
Open edX Caliper Tracking can be used to transform the edX traditional event logs intoCaliper Analytics Specificationsprovided byIMS Global. Generated logs can be consumed by any analytics application which is compatible with Caliper Standard. All the transformed logs areCaliper Certified.InstallationTo installopenedx-caliper-trackingin your Open edX instance, please add the following line to yourrequirements file. (For most Open edX installations it should be located atedx-platform/requirements/edx/base.txt).openedx-caliper-tracking==0.14.3For manual installation:pip install openedx-caliper-trackingConfigurationTo enable and useopenedx-caliper-tracking:AddENABLE_EVENT_CALIPERIZATIONflag underFEATURESin the following files:lms.env.jsoncms.env.jsonThese files should be located at/edx/app/edxapp/directory, see the example below:"FEATURES": {
    ...

    "ENABLE_EVENT_CALIPERIZATION": true,

    ...
}2. Add the following lines of code:if FEATURES.get('ENABLE_EVENT_CALIPERIZATION'):
    INSTALLED_APPS.insert(
        INSTALLED_APPS.index('eventtracking.django.apps.EventTrackingConfig'),
        'openedx_caliper_tracking'
    )in the following files:lms/envs/production.py (aws.py for hawthorn release)cms/envs/production.py (aws.py for hawthorn release)Note:Must make sure that after doing all the required changes inenv(lms.env.jsonandcms.env.json) andauth(lms.auth.jsonandcms.auth.json) files you restart the server to reflect the changes that you have made.Sending logs to external APIs (Optional)There are two ways we can send caliper transformed logs to any external APIRest APIKafkaUsing REST APITo do this, we have to add the following configurationsAddENABLE_CALIPER_EVENTS_DELIVERYflag underFEATURESin the following files:lms.env.jsoncms.env.jsonThese files should be located at/edx/app/edxapp/directory, see the example below:"FEATURES": {
    ...

    "ENABLE_CALIPER_EVENTS_DELIVERY": true,

    ...
}Add the keyCALIPER_DELIVERY_ENDPOINTand its value in theenvfiles (lms.env.jsonandcms.env.json).Add the keyCALIPER_DELIVERY_AUTH_TOKENand its value in theauthfiles (lms.auth.jsonandcms.auth.json).Add the following lines of code:if FEATURES.get('ENABLE_CALIPER_EVENTS_DELIVERY'):
    CALIPER_DELIVERY_ENDPOINT = ENV_TOKENS.get('CALIPER_DELIVERY_ENDPOINT')
    CALIPER_DELIVERY_AUTH_TOKEN = AUTH_TOKENS.get('CALIPER_DELIVERY_AUTH_TOKEN')in the following files:lms/envs/production.py (aws.py for hawthorn release)cms/envs/production.py (aws.py for hawthorn release)Using Kafka Broker APITo do this, we have to add the following configurationsAddENABLE_KAFKA_FOR_CALIPERflag underFEATURESin the following files:lms.env.jsoncms.env.jsonThese files should be located at/edx/app/edxapp/directory, see the example below:"FEATURES": {
    ...

    "ENABLE_KAFKA_FOR_CALIPER": true,

    ...
}2. Add the following keys and their values in thelms.env.jsonandcms.env.jsonfiles.
Please note that all parameters in thePRODUCER_CONFIGare unique to the broker instances. You
can set whatever parameters are required for your instance."CALIPER_KAFKA_SETTINGS": {
    "PRODUCER_CONFIG": {
        "bootstrap_servers": [
            "<List of Kafka Brokers URLs>"
        ],
        ...
    },

    "TOPIC_NAME": "<Kafka Topic Name>",

    "ERROR_REPORT_EMAILS": [
        "<Reporting Email Address 1>",
        "<Reporting Email Address 2>",
        ...
    ]
    "MAXIMUM_RETRIES": <An Integer>
},KeysDescriptionMAXIMUM_RETRIESNumber of times the app will try to send the logs to Kafka in case of failurePRODUCER_CONFIGConfigurations for initializing the Kafka ProducerCan further contain:“bootstrap_servers”:List of Kafka Brokers URLsAny other supported paramter in theKafka-python docsPlease note that it’s better to store the sensitive information in
the*.auth.jsonfilesTOPIC_NAMETopic name for the Kafka brokerERROR_REPORT_EMAILSEmail Addresses to notify when number of failures exceeds the MAXIMUM_RETRIES3. Add the following keys and their values in thelms.auth.jsonandcms.auth.jsonfiles.
Please note that all parameters in thePRODUCER_CONFIGare unique to the broker instances. You
can set whatever parameters are required for you."CALIPER_KAFKA_AUTH_SETTINGS": {
    "PRODUCER_CONFIG": {
        ...
        "sasl_plain_username": "<Username>",
        "sasl_plain_password": "<Password>",
        "security_protocol": "<Secuirty Protocol>",
        "ssl_cafile": "<Path/to/the/ca/file>",
        ...
    }
}KeysDescriptionPRODUCER_CONFIGConfigurations for initializing the Kafka Producer. Use this confiration to
store all sensitive configuration like authentication parameters.For example:Use this to configure paramters like:sasl_plain_usernamesasl_plain_passwordsecurity_protocolsasl_mechanismIt can further contain:Any other supported paramter in theKafka-python docsPlease note that it’s better to store the insensitive information
in the*.env.jsonfilesAdd the following lines of code:if FEATURES.get('ENABLE_KAFKA_FOR_CALIPER'):
    CALIPER_KAFKA_SETTINGS = ENV_TOKENS.get('CALIPER_KAFKA_SETTINGS')
    CALIPER_KAFKA_AUTH_SETTINGS = AUTH_TOKENS.get('CALIPER_KAFKA_AUTH_SETTINGS')in the following files:lms/envs/production.py (aws.py for hawthorn release)cms/envs/production.py (aws.py for hawthorn release)Location of Transformed LogsNote:This doesn’t work locally.Transformed events are logged using'logging.handlers.SysLogHandler'with'facility: local2'.We need to create output files manually and set appropriate permissions for syslog user. To do so, please follow the steps below:1. Create a log file with read/write permissions given tosysloguser (e.g:/edx/var/log/caliper-analytics/caliper.log).cd /edx/var/log
mkdir -p caliper-analytics && cd caliper-analytics
touch caliper.log
chown syslog caliper.logCreate a mapping for'local2'in the configuration files present in/etc/rsyslog.d/(e.g: in99-edx.conf).local2.*                 /edx/var/log/caliper-analytics/caliper.log;trackingRun the following command on server to restart the rsyslog daemon:sudo service rsyslog restartLocation of Logs Whose Delivery to Kafka is failedNote:This doesn’t work locally. Do this only if you are sending logs to external source using Kafka broker API.Transformed events are logged using'logging.handlers.SysLogHandler'with'facility: local3'.We need to create output files manually and set appropriate permissions for syslog user. To do so, please follow the steps below:1. Create a log file with read/write permissions given tosysloguser (e.g:/edx/var/log/caliper-analytics/delivery_failure.log).cd /edx/var/log
mkdir -p caliper-analytics && cd caliper-analytics
touch delivery_failure.log
chown syslog delivery_failure.logCreate a mapping for'local3'in the configuration files present in/etc/rsyslog.d/(e.g: in99-edx.conf).local3.*                 /edx/var/log/caliper-analytics/delivery_failure.log;trackingRun the following command on server to restart the rsyslog daemon:sudo service rsyslog restartRunning Tests LocallyTo run the unit tests of this app locally, follow the following steps:Clone the repositorygit clone git@github.com:ucsd-ets/openedx-caliper-tracking.gitRun the following command in the same directory in which you have cloned the repositorysudo ./openedx-caliper-tracking/openedx_caliper_tracking/tests/local_test_script.shLicenseThe code in this repository is licensed under the GPL v3.0 unless otherwise noted. Please seeLICENSEfor details.How To ContributeTo contribute, please make a pull request in therepositoryon Github . If you have any questions or issues, please feel free to open an issue on Github:Open edX Caliper Tracking.ContributorsMuhammad ZeeshanTasawer NawazAroosha ArifOsama ArshadDanial MalikHamza FarooqHassan TariqMuhammad Umar KhanTehreem SadatMuhammad ArslanSaad AliHusnain Raza Ghaffar