Package: edx-django-utils
edx-django-utilsEdX utilities for Django Application development.Note that some utilities may warrant their own repository. A judgement call
needs to be made as to whether code properly belongs here or not. Please
review with the Architecture Team if you have any questions.PurposeThis repository includes shared utilities for:Cache Utilities: Includes a RequestCache and a TieredCache.Django User and Group Utilities: Includes user and group utilities.IP Address Utilities: Utilities for handling request IP addresses.Logging Utilities: Includes log filters and an encrypted logging helper.Monitoring Utilities: Includes Middleware and utilities for enhanced monitoring.
At this time, supports NewRelic monitoring.Plugin Infrastructure: Enables enhanced Django Plugin capabilities.Security Utilities: Includes a middleware to add CSP response headers.Data Generation: Management command for generating Django data based on model factories.DocumentationThe full documentation is in the docs directory, and is published tohttps://edx-django-utils.readthedocs.org.Getting Started with DevelopmentPlease see the Open edX documentation forguidance on Python developmentin this repo.Design Pattern followed by packagesAll tools in edx_django_utils should expose their public api in their __init__.py files. This entails adding to __init__.py all functions/classes/constants/objects that are intended to be used by users of library.Getting HelpIf you’re having trouble, we have discussion forums atdiscuss.openedx.orgwhere you can connect with others in the
community.Our real-time conversations are on Slack. You can request aSlack
invitation, then join ourcommunity Slack workspace.For anything non-trivial, the best path is toopen an issuein this
repository with as many details about the issue you are facing as you
can provide.For more information about these options, see theGetting Helppage.How To ContributeContributions are very welcome.Please readHow To Contributefor details.PR description template should be automatically applied if you are sending PR from github interface; otherwise you
can find it it atPULL_REQUEST_TEMPLATE.mdIssue report template should be automatically applied if you are sending it from github UI as well; otherwise you
can find it atISSUE_TEMPLATE.mdThis project is currently accepting all types of contributions, bug fixes, security fixes, maintenance work, or new features. However, please make sure to have a discussion about your new feature idea with the maintainers prior to beginning development to maximize the chances of your change being accepted. You can start a conversation by creating a new issue on this repo summarizing your idea.Open edX Code of ConductAll community members are expected to follow theOpen edX Code of Conduct.PeopleThe assigned maintainers for this component and other project details may be
found inBackstage. Backstage pulls this data from thecatalog-info.yamlfile in this repo.Reporting Security IssuesPlease do not report security issues in public. Please emailsecurity@openedx.org.LicenseThe code in this repository is licensed under the Apache License, Version 2.0, unless
otherwise noted.Please seeLICENSE.txtfor details.Change Log[5.10.1] - 2024-01-17AddedAdded manufacture_data management command[5.9.0] - 2023-11-27RemovedRemovededx_django_utils.cache.disable_forced_cache_miss_for_nonewhich was added in5.7.0.[5.8.0] - 2023-11-03ChangedAdjustedget_plugin_appsto log at info level rather than debug and with more detail, though with a comment that this may not actually end up logging.[5.7.0] - 2023-08-04AddedSupport added for Django 4.2FixedFixed bug where None was not properly being stored by TieredCache.
For backward compatibility,edx_django_utils.cache.disable_forced_cache_miss_for_nonewaffle switch has
been added, which defaults to the old broken behavior of treating None as a cache miss.[5.6.0] - 2023-07-24ChangedUpdated and renamed new_relic_nrql_search to search in text widgets as well as NRQL queries[5.5.0] - 2023-06-01ChangedSwitched tosphinx-book-themeas the new standard theme across all Open
edX repos.  Seehttps://github.com/openedx/edx-sphinx-theme/issues/184for
more details.CookieMonitoringMiddleware will now remove cookies based on aCOOKIE_PREFIXES_TO_REMOVEsetting[5.4.0] - 2023-04-12AddedAdded Content-Security-Policy response header middleware undersecurity/csp[5.3.0] - 2023-03-17FixedReport both specified and existing email when refusing to create a user[5.2.0] - 2022-10-06AddedAdded a wrapper for background_task in monitoring.[5.1.0] - 2022-09-19AddedUtilities for safely determining the IP address of a request:edx_django_utils.ip(moved from edx-platform)[5.0.1] - 2022-09-09ChangedLicense has been changed from AGPL v3 to Apache v2 to reflect existing policies[5.0.0] - 2022-05-19ChangedCorrupt cookie logging:Make independent of other cookie logging; no longer needs to meet cookie size threshold or sampling rate.Breaking change, although low impact: Setting name changed fromUNUSUAL_COOKIE_SAMPLING_PUBLIC_KEYtoUNUSUAL_COOKIE_HEADER_PUBLIC_KEY.New settingUNUSUAL_COOKIE_HEADER_LOG_CHUNKhelps avoid truncated (non-decryptable) messages by splitting the output across multiple log messages.[4.8.1] - 2022-05-06AddedAdded ability to log headers when a corrupted cookie is detected in a large (or sampled) cookie header[4.8.0] - 2022-05-06Bad version – tag does not match package version, not released to PyPI. Released as 4.8.1 instead.[4.7.0] - 2022-05-05AddedAddedencrypt_for_loglogging helper andlog-sensitiveCLI command for encrypted logging of sensitive information[4.6.0] - 2022-03-16AddedAddedCookieMonitoringMiddlewarefor monitoring cookie header sizes and cookie sizes.[4.5.0] - 2022-01-31RemovedRemoved Django22, 30 and 31 from CI[4.4.2] - 2022-01-24FixedNo longer clear theRequestCacheduring the exception-handling phase (wait until response phase)It turns out all theprocess_exceptionmethods get called until one returns a response, and onlythendo theprocess_responsemethods start getting called. The result was that on exception, some middlewares were unable to use RequestCache’d values in their response phase.UpdatedReplaced usage of ‘django.conf.urls’ with ‘django.urls’[4.4.1] - 2021-12-17UpdatedReplaced usage of ‘django.conf.urls.url()’ with ‘django.urls.re_path()’[4.4.0] - 2021-09-02AddedAddedDeploymentMonitoringMiddlewareto recordPythonandDjangoversions in NewRelic with each transaction.[4.3.0] - 2021-07-20AddedAdded user and group management utilities.[4.2.0] - 2021-07-14AddedAdded support for Django 3.1 and 3.2[4.1.0] - 2021-06-01AddedAdded mixin for a custom Django admin class which disables CRUD operation on the admin’s model.AddedScript new_relic_nrql_search.py to search the NRQL in New Relic alert policies and dashboards using a supplied regex.[4.0.0] - 2021-05-03RemovedRemoved the old location ofCodeOwnerMonitoringMiddleware. It had moved in a past commit. Although technically a breaking change, all references in the Open edX platform have already been updated to point to the new location.AddedAdded newcode_owner_themeandcode_owner_squadcustom attributes. This is useful in cases where thecode_ownercombines a theme and squad name, because monitoring can instead referencecode_owner_squadto be resilient to theme name updates. For the decision doc, see edx_django_utils/monitoring/docs/decisions/0004-code-owner-theme-and-squad.rst.UpdatedMisconfigurations of CODE_OWNER_MAPPINGS will now fail fast, rather than just logging. Although technically a breaking change, if CODE_OWNER_MAPPINGS is in use, it is probably correctly configured and this change should be a no-op.[3.16.0] - 2021-03-24AddedAddedpluggable_overridedecorator.[3.15.0] - 2021-03-02Added chunked_queryset utility.[3.14.0] - 2020-12-15RemovedDropped support for Python 3.5.[3.13.0] - 2020-11-18AddedAdded record_exception to monitor caught exceptions.UpdatedAdded additional details to thedeprecated_monitoring_utilscustom attribute values to make it simpler to track down usage.[3.12.0] - 2020-11-17AddedAdded set_code_owner_attribute decorator for use with celery tasks.Added set_code_owner_attribute_from_module as an alternative to the decorator.UpdatedCleaned up some of the code owner middleware code. In doing so, renamed custom attribute code_owner_path_module to code_owner_module. This may affect monitoring dashboards. Also slightly changed when error custom attributes are set.[3.11.0] - 2020-10-31AddedAdded ADR 0004-public-api-and-app-organization.rst to explain a new app organization, which makes use of the public API more consistent.UpdatedApplied the new app organization described in th ADR to the monitoring Django app.Moved CachedCustomMonitoringMiddleware, CodeOwnerMonitoringMiddleware, and MonitoringMemoryMiddleware to the public API.DeprecatedDeprecated the old locations of CachedCustomMonitoringMiddleware, CodeOwnerMonitoringMiddleware, and MonitoringMemoryMiddleware.Deprecated various methods from modules that were always meant to be used from the public API.accumulateincrementset_custom_attributeset_custom_attributes_for_course_keyAdded additional custom attributes for deprecated classes and methods to make them safer to retire.NoteSome method implementations that were available in the public API were moved without adding a deprecated equivalent. These were not found when searching, so hopefully they are only used via the public API, which did not change. This includes functions intransactions.pyandcode_owner/utils.py.RemovedRemoved the middleware ordering checks. This is not a typical Django feature and it is painful when refactoring.[3.10.0] - 2020-10-28AddedAdded logging filter classes for users and remote IP addresses to be used by all IDAs. These were moved here from edx-platform.[3.9.0] - 2020-10-21UpdatedExposed existing get_code_owner_from_module via the public api.Fixed get_code_owner_from_module to not require a call to is_code_owner_mappings_configured beforehand.Set the existing code_owner_path_module custom attribute, even for cases where the transaction name was used, rather than the view module.Refactor code owner setting processing.[3.8.0] - 2020-08-31UpdatedRenamed “custom metric” to “custom attribute” throughout the monitoring library. This decision can be read about in the ADR 0002-custom-monitoring-language.rst.  The following have been deprecated:set_custom_metric (use set_custom_attribute)set_custom_metrics_for_course_key (use set_custom_attributes_for_course_key)MonitoringCustomMetricsMiddleware (use CachedCustomMonitoringMiddleware)CachedCustomMonitoringMiddleware.accumulate_metric (use CachedCustomMonitoringMiddleware.accumulate_attribute)This wasn’t meant to be used publicly, but was deprecated just in case.CodeOwnerMetricMiddleware (use CodeOwnerMonitoringMiddleware)[3.7.4] - 2020-08-29Fix to custom monitoring accumulate to actually accumulate rather than overwrite.[3.7.3] - 2020-08-12UpdatedUpgrade psutil to latest version[3.7.2] - 2020-08-10UpdatedAdded missing classes to plugins public api. Seeplugins.__init__.pyfor latest api.Updated plugin method names to be more descriptive. Seeplugins.__init__.pyfor latest.NoteAlthough these changes are backwards incompatible, they are being added as a bug fix because plugins code release (3.7.0) is not yet in use.[3.7.1] - 2020-08-10UpdatedExposing all public functions in edx_django_utils/plugins directory in its __init__.py file.this was done to keep inline with standard/pattern used in other packages in edx_django_utils[3.7.0] - 2020-08-10AddedAdding Plugin infrastructureAllows IDAs to use plugins[3.6.0] - 2020-08-04AddedImproved documentation for CodeOwnerMetricMiddleware, including a how_tos/add_code_owner_custom_metric_to_an_ida.rst for adding it to a new IDA.Added ignore_transaction monitoring utility to ignore transactions we don’t want tracked.UpdatedMoved transaction-related monitoring code into it’s own file. Still exposed through__init__.pyso it’s a non-breaking change.[3.5.0] - 2020-07-22UpdatedAdded a catch-all capability to CodeOwnerMetricMiddleware when CODE_OWNER_MAPPINGS includes a ‘*’ as a team’s module. The catch-all is used only if there is no other match.[3.4.0] - 2020-07-20AddedAdded get_current_transaction for monitoring that returns a transaction object with a name property.UpdatedUpdated CodeOwnerMetricMiddleware to use NewRelic’s current transaction for cases where resolve() doesn’t work to determine the code_owner, like for Middleware.[3.3.0] - 2020-07-16AddedCodeOwnerMetricMiddleware was moved here (from edx-platform) in order to be able to take advantage of thecode_ownermetric in other IDAs. For details on this decision, see theADR for monitoring code owner. See the docstring for more details on usage.[3.2.3] - 2020-05-30Removed ceninusepy3 usage.[3.2.2] - 2020-05-04Added support for python 3.8 and dropped support for Django versions older than 2.2[3.2.1] - 2020-04-17Changedimported get_cache_key in cache/__init__.py.[3.2.0] - 2020-04-09AddedAdded get_cache_key utility.[2.0.1] - 2019-10-09ChangedFixed: Updated function tracing to accomodate changes in New Relic’s 5.x Agent.[2.0.0] - 2019-07-07ChangedConverted Middleware (from old style MIDDLEWARE_CLASSES to MIDDLEWARE).Removed support for Django versions < 1.11[1.0.1] - 2018-09-07ChangedFixed: RequestCache now properly uses thread.local.Fixed: CachedResponse.__repr__ now handles unicode.[1.0.0] - 2018-08-28AddedAdddatadict property to better match legacy RequestCache interface.ChangedChange is_hit/is_miss to is_found.[0.5.1] - 2018-08-17ChangedFixed bug in TieredCacheMiddleware dependency declaration.[0.5.0] - 2018-08-16ChangedRestored Python 3 support.Refactor/clean-up, including Middleware dependency checking.Docs updates and other cookiecutter updates.[0.4.1] - 2018-08-10ChangedSplit out TieredCacheMiddleware from RequestCacheMiddleware.[0.4.0] - 2018-08-10ChangedRename CacheUtilsMiddleware to RequestCacheMiddleware.[0.3.0] - 2018-08-02RemovedTemporarily dropped Python 3 support to land this.[0.2.0] - 2018-08-01AddedAdded cache and monitoring utilities.[0.1.0] - 2018-07-23AddedFirst release on PyPI.