Package: occupationcoder
occupationcoderA tool to assign standard occupational classification codes to job vacancy descriptionsGiven a job title, job description, and job sector the algorithm assigns
a UK 3-digit standard occupational classification (SOC) code to the job.
The algorithm uses theSOC 2010standard, more details of which can
be found onthe ONS’
website.This code originally written by Jyldyz Djumalieva,Arthur
Turrell, David Copple, James
Thurgood, and Bradley Speigner. If you use this code please cite:Turrell, A., Speigner, B., Djumalieva, J., Copple, D., & Thurgood, J.
(2019).Transforming Naturally Occurring Text Data Into Economic
Statistics: The Case of Online Job Vacancy
Postings(No. w25837). National
Bureau of Economic Research.@techreport{turrell2019transforming,
  title={Transforming naturally occurring text data into economic statistics: The case of online job vacancy postings},
  author={Turrell, Arthur and Speigner, Bradley and Djumalieva, Jyldyz and Copple, David and Thurgood, James},
  year={2019},
  institution={National Bureau of Economic Research}
}Documentation:https://occupationcoder.readthedocs.io.Pre-requisitesSeesetup.pyfor a full list of Python packages.occupationcoder is built on top ofNLTKand
uses ‘Wordnet’ (a corpora, number 82 on their list) and the Punkt
Tokenizer Models (number 106 on their list). When the coder is run, it
will expect to find these in their usual directories. If you have nltk
installed, you can get them corpora usingnltk.download()which will
install them in the right directories or you can go tohttp://www.nltk.org/nltk_data/to
download them manually (and follow the install instructions).A couple of the other packages, such asfuzzywuzzydo not come
with the Anaconda distribution of Python. You can install these via pip
(if you have access to the internet) or download the relevant binaries
and install them manually.File and folder descriptionoccupationcoder/coderapplies SOC codes to job descriptionsoccupationcoder/createdictionariesturns the ONS’ index of SOC
code into dictionaries used byoccupationcoder/coderoccupationcoder/dictionariescontains the dictionaries used byoccupationcoder/coderoccupationcoder/outputsis the default output directoryoccupationcoder/testvacanciescontains ‘test’ vacancies to run
the code onoccupationcoder/utilitiescontains helper functions which mostly
manipulate stringsInstallation via terminal using pipDownload the package and navigate to the download directory. Then usepythonsetup.pysdistcddistpipinstalloccupationcoder-version.tar.gzThe first line creates the .tar.gz file, the second navigates to the
directory with the packaged code in, and the third line installs the
package. The version number to use will be evident from the name of the
.tar.gz file.Running the code as a python scriptImporting, and creating an instance, of the coderimportpandasaspdfromoccupationcoder.coderimportcodermyCoder=coder.Coder()To run the code on a single job, use the following syntax with thecodejobrow(job_title,job_description,job_sector)method:if__name__=='__main__':myCoder.codejobrow('Physicist','Calculations of the universe','Professional scientific')Theifstatement is required because the code is parallelised. Note
that you can leave some of the fields blank and the algorithm will still
return a SOC code.To run the code on a file (eg csv name ‘job_file.csv’) with structurejob_titlejob_descriptionjob_sectorPhysicistMake calculations about the universe, do research, perform experiments and understand the physical environment.Professional, scientific & technical activitiesusedf=pd.read_csv('path/to/foo.csv')df=myCoder.codedataframe(df)This will return a new dataframe with SOC code entries appended in a new
column:job_titlejob_descriptionjob_sectorSOC_codePhysicistMake calculations about the universe, do research, perform experiments and understand the physical environment.Professional, scientific & technical activities211Running the code from the command lineIf you have all the relevant packages in requirements.txt, download the
code and navigate to the occupationcoder folder (which contains the
README). Then runpython-moccupationcoder.coder.coderpath/to/foo.csvThis will create a ‘processed_jobs.csv’ file in the outputs/ folder
which has the original text and an extra ‘SOC_code’ column with the
assigned SOC codes.TestingTo run the tests in your virtual environment, usepython-munittestin the top level occupationcoder directory. Look intest_occupationcoder.pyfor what is run and examples of use. The output appears in the ‘processed_jobs.csv’ file in the outputs/
folder.AcknowledgementsWe are very grateful to Emmet Cassidy for testing this algorithm.DisclaimerThis code is provided ‘as is’. We would love it if you made it better or
extended it to work for other countries. All views expressed are our
personal views, not those of any employer.CreditsThe development of this package was supported by the Bank of England.This package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.2.0 (2021-04-15)First release on PyPI.