Package: opportunity-scraper
opportunity_scraperopportunity_scraperis a Python3 script for scraping R&D competition websites and dumping the results into the SuiteCRM V8 API.InstallationInstall withpip:pip3installopportunity_scraperDependenciesopportunity_scraperuseschromedriver, and will not work without it. Download the appropriatechromedriverversion for your Chrome or Chromium version. Some distros (Fedora, at the very least) may have a packaged version that corresponds to the versions of Chrome available.Supported resourcesUK Government Innovation Funding ServiceKTN Innovation Exchange(requires registration)Before useSuiteCRM OAuth2 credentialsBefore using the scraper, you need to create some SuiteCRM OAuth2 client credentials for the script to use.Clickfor the official SuiteCRM documentation. It's a little confusing on what to do, so for clarity:Navigate to theOAuth2 Clients and Tokensadmin page (https://www.your-suitecrm-instance.com/suitecrm/index.php?module=OAuth2Clients)Create a newClient Credentials Client, by clicking "New Client Credentials Client", giving it a name, and entering a secret password in the input box labelled "Change secret". Despite the wording, do not leave the box blank when creating your credentials.After saving, you'll be presented with a Client ID.The Client ID and Client Secret will be used in configuration.ConfigurationConfiguration is via aYAMLfile.The default location is~/.config/opportunity_scraper/settings.yaml, but this can be changed with the--configCLI option at runtime.The following can be used as a template for the config file. All values are required.browser:# Location of chromedriverchromedriver:"/usr/bin/chromedriver"# Credentials for KTN Innovation Exchangektn_username:"dummy@example.com"ktn_password:"somesecretpassword"oauth:# SuiteCRM OAuth2 credentialstoken_url:"https://example.com/suitecrm/Api/access_token"client_id:"some-uuid-token"client_secret:"anotherpassword"suitecrm:# SuiteCRM APIapi_url:"https://example.com/suitecrm/Api"# Sales Account ID for .gov.uk competitionsgovuk_account_id:"another-uuid"# Sales Account ID for KTN competitionsktn_account_id:"another-uuid"# Default user to be assigned new opportunitiesassigned_user_id:"1"Notes on configurationbrowserchromedriver: path to thechromedriverexecutable. Depending on whether you install it from a distro package or the Chromedriver website, it may or may not end up in$PATH, so for simplicity we specify it here.ktn_username,ktn_password: KTN Innovation Exchange credentials, which can be createdhere.oauthtoken_url: Full URL to the SuiteCRM V8 API OAuth2 endpoint - usually of the formhttps://your-suitecrm-instance.com/suitecrm/Api/access_tokenclient_id,client_secret: These must be created in the SuiteCRM admin, as above.suitecrmapi_url: Full URL to the SuiteCRM V8 API - usually of the formhttps://your-suitecrm-instance.com/suitecrm/Api.govuk_account_id,ktn_account_id: The ID (in UUID form) of the SuiteCRM Sales Account that will be assigned to new opportunities scraped from the gov.uk and KTN sites. The UUID can be found in the URL when examining a particular Sales Account in SuiteCRM (https://your-suitecrm-instance.com/suitecrm/index.php?module=Accounts&action=index).The same SuiteCRM account can be used for .gov.uk- and KTN-sourced opportunities if desired - just use the same ID for both values.Runningusage: opportunity_scraper [-h] [-c CONFIG]

Scrape R&D competitions and push the results to the SuiteCRM API.

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        Location of config fileLicenseGPLv3