Package: scrape-jobs
Simple CLI jobs scraperDISCLAIMERMade as POCUSE AT YOUR OWN RISKWorkflowScrape jobs matching certain criterialinkedin.com - [ keywords , location ]seek.com.au - [ what , where ]Store the scraped dataupload to Google Sheets.or just store it locally to CSV file.InstallationPrerequisites:Google Chrome installedchromedriver binary executable in PATHPython 3.6+Prepared Google Spreadsheet (detailed instructions bellow)Install command:pip install –force -U scrape-jobsInstalls the following CLI bindings:scrape-jobs-init-configscrape-jobsBasic InstructionsOpen terminal/CMD and ‘cd’ into working dir.Runscrape-jobs-init-configto populate sample config file in the current work dirusage: scrape-jobs-init-config [-h] [–version] [-v] [-vv] [-f FILE]initialize sample ‘scrape-jobs’ config fileoptional arguments:-h,--helpshow this help message and exit--versionshow program’s version number and exit-v,--verboseset loglevel to INFO-vv,--very-verboseset loglevel to DEBUG-fFILEdefaults to ‘/current/work/dir/scrape-jobs.ini’Edit the config file as per your needsRunscrape-jobsto trigger executionusage: scrape-jobs [-h] [–version] [-v] [-vv] [-c CONFIG_FILE]{linkedin.com,seek.com.au}Scrape jobs and store results.positional arguments:{linkedin.com,seek.com.au}site to scrapeoptional arguments:-h,--helpshow this help message and exit--versionshow program’s version number and exit-v,--verboseset loglevel to INFO-vv,--very-verboseset loglevel to DEBUG-cCONFIG_FILEdefaults to ‘/current/work/dir/scrape-jobs.ini’More Detailed Instructions:Prepare the spreadsheet and the spreadsheet’s auth.json (Spreadsheet instructions at the bottom)Forseek.com.authe Worksheet’s columns are:[“scraped_time”, “posted_time”, “location”, “area”, “classification”, “sub_classification”, “title”, “salary”, “company”, “url”]Forlinkedin.comthe Worksheet’s columns are:[“scraped_time”, “posted_time”, “location”, “title”, “company”, “url”]Init empty config file by callingscrape-jobs-init-configEdit the newly createdCWD\scrape-jobs.iniwith params of your choiceset the path to the AUTH.JSONset the spreadsheet nameset the worksheet name  (it will be automatically created if it doesn’t exist)set the search paramsTrigger execution:runscrape-jobs linkedin.comorscrape-jobs seek.com.auyou will see output in the console, but a scrape-jobs.log will be created tooto have more detailed output add-vvexecution paramAfter the scrape is complete you should see the newly discovered jobs in your spreadsheetAlternatively you can init a config at a known place and just pass it’s path:scrape-jobs-init-config -f /custom/path/to/config.iniscrape-jobs -c /custom/path/to/config.ini seek.com.auNoteYou need to prepare AUTH.JSON file in advance that is to be used for authentication with GoogleSheetsThe term ‘Spreadsheet’ refers to a single document that is shown in the GoogleSpreadsheets landing pageA single ‘Spreadsheet’ can contain one or more ‘Worksheets’Usually a newly created ‘Spreadsheet’ contains a single ‘Worksheet’ named ‘Sheet1’If you don’t provide a valid path to AUTH.JSON the collected data will be saved as .csv in the current work dirInstructions for preparing Google Spreadsheet AUTH.JSON:Go tohttps://console.developers.google.com/Login with the google account that is to be owner of the ‘Spreadsheet’.At the top-left corner, there is a drop-down right next to the “Google APIs” textClick the drop-down and a modal-dialog will appear, then click “NEW PROJECT” at it’s top-rightName the project relevant to how the sheet is to be used, don’t select ‘Location*’, just press ‘CREATE’Open the newly created project from the same drop-down as in step 3.There should be ‘APIs’ area with a “-> Go to APIs overview” at it’s bottom - click itA new page will load having ‘+ ENABLE APIS AND SERVICES’ button at the top side’s middle - click itA new page will load having a ‘Search for APIs & Services’ input - use it to find and open ‘Google Drive API’In the ‘Google Drive API’ page click “ENABLE” - you’ll be redirected back to the project’s pageThere will be a new ‘CREATE CREDENTIALS’ button at the top - click itSetup the new credentials as follows:Which API are you using? -> ‘Google Drive API’Where will you be calling the API from? -> ‘Web server (e.g. node.js, Tomcat)What data will you be accessing? -> ‘Application data’Are you planning to use this API with App Engine or Compute Engine? -> No, I’m not using them.Click the blue button ‘What credentials do I need’, will take you to ‘Add credentials to you project’ pageSetup the credentials as follows:Service account name:  {whatever name you type is OK, as long the input accepts it}Role: Project->EditorKey type: JSONPress the blue ‘Continue’ button, and a download of the AUTH.JSON file will begin (store it safe)Close the modal and go back to the project ‘Dashboard’ using the left-side navigation panelRepeat step 8.Search for ‘Google Sheets API’, then open the result and click the blue ‘ENABLE’ buttonOpen the downloaded auth.json file and copy the value of the ‘client_email’Using the same google account as in step 2. , go to the normal google sheets and create & open the ‘Spreadsheet’do a final renaming to the spreadsheet now to avoid issues in future‘Share’ the document with the email copied in step 19., giving it ‘Edit’ permissionsyou might want to un-tick ‘Notify people’ before clicking ‘Send’ as it’s a service email you’re sharing with‘Send’ will change to ‘OK’ upon un-tick, but we’re cool with that - just click it.You are now ready to use this class for retrieving ‘Spreadsheet’ handle in the code!