Package: taxonomy-connector
TaxonomyThe taxonomy service is a library that can be installed in other edX components
that provides access to third party taxonomy vendors. EMSI is currently the
only vendor available, but others may be integrated as necessary. This service
can communicate with the vendor to get job, skill, and salary data. This service
can also be used to submit data (course descriptions, etc.) to the vendor to
produce potential matches for a skill or job.After submitting a pull request, please use the Github “Reviewers” widget to add
relevant reviewers and track review process.Getting StartedTo installtaxonomy-connector, for example, in Course Discovery, follow these steps:It is recommended that you clone this repo into a sub-folder of your working directory. Create a sub-folder with namesrcif it doesn’t already exist.Clone this repository into thesrcfolder.Go to the shell of the host environment where you want to install this package and run thispip install-e/edx/src/taxonomy-connectorChanges made into the taxonomy repository will now be picked up by your host environment.Notes:In order to communicate with EMSI service, you need to set the values ofclient_idandclient_secret. These values are picked up from the host environment so you need to pass them in.yamlfile of the host environment.Also, to make taxonomy work, the host platform must add an implementation of data providers written in./taxonomy/providersTaxonomy APIs use throttle rate set inDEFAULT_THROTTLE_RATESsettings by default. Custom Throttle rate can by set by addingScopedRateThrottleclass inDEFAULT_THROTTLE_CLASSESsettings andtaxonomy-api-throttle-scopekey inDEFAULT_THROTTLE_RATESFor the skill tags to be verified, the management commandfinalize_xblockskill_tagsneeds to be run periodically.Also, You can configure the skill tags verification by setting the values ofSKILLS_VERIFICATION_THRESHOLD,SKILLS_VERIFICATION_RATIO_THRESHOLD,SKILLS_IGNORED_THRESHOLDandSKILLS_IGNORED_RATIO_THRESHOLDin the host platform or by passing the values to the command using the args--min-verified-votes,--ratio-verified-threshold,--min-ignored-votesand--ratio-ignored-threshold.REST_FRAMEWORK={'DEFAULT_THROTTLE_CLASSES':('rest_framework.throttling.UserRateThrottle','rest_framework.throttling.ScopedRateThrottle'),'DEFAULT_THROTTLE_RATES':{'user':'100/hour','taxonomy-api-throttle-scope':'60/min',# custom throttle rate for taxonomy api},}Developer NotesTo run unit tests, create a virtualenv, install the requirements withmake requirementsand then runmake testTo update the requirements, runmake upgradeTo run quality checks, runmake qualityPlease do not import models directly in course discovery. e:g if you want to import CourseSkills in Discovery, use the utility get_whitelisted_course_skills instead of directly importing it.Reporting Security IssuesPlease do not report security issues in public. Please emailsecurity@openedx.org.Getting HelpHave a question about this repository, or about Open edX in general?  Please
refer to thislist of resourcesif you need any assistance.Change LogUnreleased[1.46.2] - 2024-02-14feat: Optimized finalize_xblockskill_tags command for memory via chunking[1.46.1] - 2024-01-05feat: Modify prefetch related to select related for whitelisted product skills.[1.46.0] - 2023-10-23feat: Removed direct usages of JobSkills and IndustryJobSkills objects in favour of whitelisted or blacklisted query sets.[1.45.0] - 2023-10-13feat: Added the ability to blacklist job-skill relationship.[1.44.3] - 2023-09-20perf: improve xblock skills api performance.[1.44.2] - 2023-09-11fix: chunked data at 50000 byte in EMSI client for xblock-skills job[1.44.1] - 2023-08-25feat: add prefetch related to the whitelisted product skills[1.44.0] - 2023-08-09feat: Added the ability to ignore unrelated jobs from being indexed on algolia.[1.43.4] - 2023-08-02fix: Added missing comma in algolia constants that was masking the b2c_opt_in attribute.[1.43.3] - 2023-08-02fix: Fixed some implementation issues in job recommendation logic.[1.43.2] - 2023-08-01perf: Further performance enhancements for the algolia index command.[1.43.1] - 2023-07-31perf: Performance enhancements in job recomendations calculation.[1.43.0] - 2023-07-07feat: reuse tags from similar product for xblock skills.fix: remove duplicates from xblocks skills api.refactor: update logic to mark course run complete in refresh_xblock_skills command.[1.42.3] - 2023-07-14perf: memory optimisation for job recommendations.[1.42.2] - 2023-07-14perf: pandas dataframe loading memory optimisation[1.42.1] - 2023-07-06fix: Use autocomplete field for selecting job in B2CJobAllowlist.[1.42.0] - 2023-07-04feat: Added a new fieldjob_sourcesin job’s algolia index.[1.41.0] - 2023-06-14feat: change B2C Job Allowlist to admin config.[1.40.1] - 2023-06-06feat: Added pagination in JobHolderUsernamesAPIView API.[1.40.0] - 2023-06-02feat: Added JobHolderUsernamesAPIView to fetch current job of learners by LMS.[1.39.0] - 2023-05-09feat: Added CourseRunMetadataProvider to fetch course run info.feat: Added CourseRunXBlockSkillsTracker model to track xblock tagging under courses.Switch fromedx-sphinx-themetosphinx-book-themesince the former is
deprecated.  Seehttps://github.com/openedx/edx-sphinx-theme/issues/184for
more details.[1.38.1] - 2023-05-11fix: Update theattributesForFacetinglist to include theb2c_opt_infield[1.38.0] - 2023-05-03feat: Added a new attribute (b2c_opt_in) to the JobSerializer[1.37.3] - 2023-05-03feat: generate job description only if job has name and description is empty[1.37.2] - 2023-04-27feat: Generate ai based job descriptions[1.37.1] - 2023-03-31feat: making sub_category skills to job specific in career tab.[1.37.0] - 2023-03-31Added the ability to remove unused jobs from django admin.[1.36.3] - 2023-03-29fix: Do not create a job if all of the releated skills does not exist in database[1.36.2] - 2023-03-08fix: remove validations on skills in skill quiz[1.36.1] - 2023-02-23Index industry data with skills in Algolia.[1.36.0] - 2023-02-20Added handler for openedx-events: XBLOCK_SKILL_VERIFIED.[1.35.1] - 2023-02-10Enabled ordering in SkillsQuizViewSet.[1.35.0] - 2023-02-07Added logic to avoid 429 errors and handle these errors if they still appear while communicating with LightCast API.[1.34.0] - 2023-01-10Added similar jobs list in jobs algolia jobs index.[1.33.0] - 2023-01-09https://github.com/openedx/openedx-events/pull/143merged, so adding back
changes reverted in version 1.32.1Added refresh_xblock_skills command to update skills for xblocks.Added handlers for openedx-events: XBLOCK_DELETED, XBLOCK_PUBLISHED and XBLOCK_PUBLISHED.Added finalize_xblockskill_tags to mark skills as verified or blacklisted.[1.32.3] - 2023-01-05Added log for EMSI client access token and raising error for error status.[1.32.2] - 2023-01-02updated requirements.[1.32.1] - 2022-12-20Reverts changes depending on openedx-events till upstream MR is merged.https://github.com/openedx/openedx-events/pull/143[1.32.0] - 2022-12-20Added refresh_xblock_skills command to update skills for xblocks.[1.31.2] - 2022-12-23Added ACCESS_TOKEN_EXPIRY_THRESHOLD_IN_SECONDS in EMSI client.[1.31.1] - 2022-12-19Handle repeating industry names in algolia index and test[1.31.0] - 2022-12-06Added handlers for openedx-events: XBLOCK_DELETED, XBLOCK_PUBLISHED and XBLOCK_PUBLISHED.[1.30.1] - 2022-12-06Added xblocks to skill API.Added xblocks API.[1.30.0] - 2022-12-06Added industry_names facet in Algolia Jobs Index.[1.29.0] - 2022-11-28Added XBlockSkills and XBlockSkillData models.Added related celery tasks, abstract provider, signals and commands.Added management command to verify xblockskill tags.[1.28.2] - 2022-11-23Added industry_names field in Algolia serializer.[1.28.1] - 2022-11-22Added JobHolderUsernamesAPIView which returns a list of 100 usernames from SkillsQuiz.[1.28.0] - 2022-11-21Updated refresh_job_skills command to save industry relation with job and skills.[1.27.0] - 2022-10-31Removed industry foreign key from JobSkills table and create a new table IndustryJobSkill.[1.26.0] - 2022-10-31Added relation between JobSkill and Industry Table.[1.25.0] - 2022-10-24Added JobTopSkillCategoriesAPIView.[1.24.0] - 2022-10-21Added a new model to store industry data using NAICS2 codes.[1.23.1] - 2022-10-13Do no concatenate ifshort_description is `None.Fix CourseSkills update_or_create call.[1.23.0] - 2022-10-05Expand course skills tagging to includetitle,short_descriptionandfull_description.[1.22.5] - 2022-09-16Fixes product type issue by using ProductTypes choices.[1.22.4] - 2022-09-14Updated utils to support program skills.[1.22.3] - 2022-09-07Added support to filter Skills by names.[1.22.2] - 2022-09-06Register Program associated models on Admin.Change verbose name for RefreshProgramSkillsConfig model[1.22.1] - 2022-08-26Added id field in JobSerializer for Algolia.[1.22.0] - 2022-08-22Added a new model for storing user response for skills quiz.Added new REST endpoints for performing CRUD operations on skills quiz.[1.21.0] - 2022-08-16feat: add task to update program skills through EMSI api[1.20.0] - 2022-08-11feat: add caching toutils.get_whitelisted_serialized_skills()[1.19.0] - 2022-08-04feat: add provider and validator for Programs[1.18.0] - 2022-08-01feat: add program skill modelfeat: Update SkillSerializer to include Category and Subcategory details.[1.17.1] - 2022-07-29feat: use program update signal to call EMSI API[1.17.0] - 2022-07-15refactor: Remove EdxRestApiClient usage in taxonomy-connector[1.16.3] - 2022-06-23Added handling for None values for median salary from EMSI.[1.16.2] - 2022-06-22Fixed error causes by null values returned by EMSI API.[1.16.1] - 2022-06-15Fixed API 429 error and updated admin list display for skills for better usability.[1.16.0] - 2022-06-08Added category and subcategory for skill.[1.15.4] - 2022-04-06fix: Add limit to EMSI API calls[1.15.3] - 2022-03-11fix: Check the course description length after encoding[1.15.2] - 2022-02-18feat: Added Support for large size course description translation[1.15.1] - 2022-02-17fix: Made the median_posting_duration in JobPosting Nullable to avoid errors on jenkins.[1.15.0] - 2022-02-11chore: Removed Django22, 30 and 31 support and added support for Django40[1.14.5] - 2022-02-08feat: Added Support for course description translation[1.14.4] - 2022-01-28feat: Add Translation model[1.14.3] - 2021-10-27fix: Make job names unique and handle exception where ever job is created/updated[1.14.2] - 2021-09-08Fixed an issue that was causing an error while index jobs data to algolia.[1.14.1] - 2021-08-20Decreased skills query chunk_size from 2000 to 50 to fetch more jobs.[1.14.0] - 2021-08-16Added managment command and related code to index jobs data to algolia.[1.13.0] - 2021-08-9Added Skill, Job and JobPostings viewsets.[1.12.2] - 2021-08-5Add job posting information in utility methodget_course_jobs.[1.12.1] - 2021-08-3Add utility methodget_course_jobsto return job associated with a course.[1.12.0] - 2021-07-13Added support for django 3.1 and 3.2[1.11.2] - 2021-05-28Added utility method to return serialized course skills.[1.11.1] - 2021-04-20Fixed .rst issues in CHANGELOG.rst[1.11.0] - 2021-04-16Mention currency in median salary field and add verbose name for models.[1.10.0] - 2021-04-12Remove all the usages of oldcourse_idfield including the column definition inCourseSkillsmodel.[1.9.0] - 2021-04-12Replace the usages of oldcourse_idinCourseSkillswith the newcourse_keyfield.[1.8.0] - 2021-04-09Added a new field namedcourse_keyinCourseSkillsmodel to deprecate and replace the oldcourse_idfield in future.[1.7.0] - 2021-04-07Removed RefreshCourseSkill view.[1.6.2] - 2021-03-12Handled edge cases inrefresh_course_skillscommand.[1.6.1] - 2021-03-10Updated logging structure forrefresh_course_skillscommand.[1.6.0] - 2021-03-09Added support for –all param inrefresh_course_skillscommand to back populate data.[1.5.0] - 2021-03-04Addedpopulate_job_namescommand.[1.4.1] - 2021-02-19Added description field in Skill model and update the refresh_course_skill command to save skill description.Pinning EMSI skills API version to 7.35[1.4.0] - 2021-02-17Updated refresh_job_skill command to get jobs related only to skills that are in our systemUpdated refresh_job_postings command to get job_posting only related to job we already have in our system.Added constrains on the Job, Skill, JobPostings, CourseSkill and JobSkill table.Added migration to remove all previous taxonomy data.Added utility to chuck the queryset provided.[1.3.6] - 2021-01-29Remove caching from EMSI API client.[1.3.5] - 2021-01-27Added some utility functions for adding/removing course skills to/from blacklist.[1.3.4] - 2021-01-27More logging.[1.3.3] - 2021-01-26Improve logging.[1.3.2] - 2021-01-25Added logs for signals and tasks.[1.3.1] - 2021-01-22Added the ability to black list course skills.[1.3.0] - 2021-01-13Added JobSkills.skill column and removed JobSkills.name column.[1.2.1] - 2021-01-07Added course update signal and handler to trigger the celery taskAdded celery task to update course skillsRefactoredrefresh_course_skillsmanagement command[1.2.0] - 2020-12-24Fixed TypeError that pops up sometimes while communicating with the EMSI API.[1.1.6] - 2020-12-24Updated the README description.[1.1.5] - 2020-12-18Fixed travis issue related to PyPI upload.[1.1.4] - 2020-12-17Fixed the bug where EMSI API was returning 404 for job posting data.[1.1.3] - 2020-11-05Updating travis configuration.[1.1.2] - 2020-10-20Updating jobs-salary data’s query.[1.1.1] - 2020-10-20Updating skills-jobs data’s query.[1.1.0] - 2020-09-30Renamed main package name from taxonomy-service to taxonomy-connector.[1.0.1] - 2020-09-21Added package data, so that migrations and python packages are included in the final build.[1.0.0] - 2020-09-09Added Providers and Validators for integrations and upgraded to the first stable release.[0.1.1] - 2020-09-09Enable Travis integration[0.1.0] - 2020-08-27Added Basic skeleton and clients to call EMSI endpoint.