Package: ycombinator-scraper
YCombinator-ScraperCI/CDDocsPackageMetaYCombinator-Scraper provides a web scraping tool for extracting data fromWorkatastartupwebsite. The package uses Selenium and BeautifulSoup to navigate through the pages and extract information.Documentation:https://nneji123.github.io/ycombinator-scraperSource Code:https://github.com/nneji123/ycombinator-scraperSponsorProxycurl APIsScrape public LinkedIn profile data at scale with Proxycurl APIs.Scraping Public profiles are battle tested in court in HiQ VS LinkedIn case.GDPR, CCPA, SOC2 compliant.High rate limit - 300 requests/minute.Fast - APIs respond in ~2s.Fresh data - 88% of data is scraped real-time, other 12% are not older than 29 days.High accuracy.Tons of data points returned per profileBuilt for developers, by developers.FeaturesWeb Scraping Capabilities:Extract detailed information about companies, including name, description, tags, images, job links, and social media links.Scrape job-specific details such as title, salary range, tags, and description.Founder and Company Data Extraction:Obtain information about company founders, including name, image, description, linkedIn profile, and optional email addresses.Headless Mode:Run the scraper in headless mode to perform web scraping without displaying a browser window.Configurability:Easily configure scraper settings such as login credentials, logs directory, automatic install of webdriver based on browser withwebdriver-manager packageand using environment variables or a configuration file.Command-Line Interface (CLI):Command-line tools to perform various scraping tasks interactively or in batch mode.Data Output Formats:Save scraped data in JSON or CSV format, providing flexibility for further analysis or integration with other tools.Caching Mechanism:Implement a caching feature to store function results for a specified duration, reducing redundant web requests and improving performance.Docker Support:Package the scraper as a Docker image, enabling easy deployment and execution in containerized environments or run the prebuilt docker imagedocker pull nneji123/ycombinator_scraper.RequirementsPython 3.9+Chrome or Chromium browser installed.Installation$pipinstallycombinator-scraper$ycscraper--help#OutputYCombinator-Scraper Version 0.7.0Usage: python -m ycombinator_scraper [OPTIONS] COMMAND [ARGS]...Options:--help  Show this message and exit.Commands:loginscrape-companyscrape-foundersscrape-jobversionWith Docker$gitclonehttps://github.com/Nneji12/ycombinator-scraper
$cdycombinator-scraper
$dockerbuild-tyour_name/scraper_name.# e.g docker build -t nneji123/ycombinator_scraper .$dockerrunnneji123/ycombinator_scraperpython-mycombinator_scraper--helpDependenciesclick: Enables the creation of a command-line interface for interacting with the scraper tool.beautifulsoup4: Facilitates the parsing and extraction of data from HTML and XML in the web scraping process.loguru: Provides a robust logging framework to track and manage log messages generated during the scraping process.pandas: Utilized for the manipulation and organization of data, particularly in generating CSV files from scraped information.pathlib: Offers an object-oriented approach to handle file system paths, contributing to better file management within the project.pydantic: Used for data validation and structuring the models that represent various aspects of scraped data.pydantic-settings: Extends Pydantic to enhance the management of settings in the project.selenium: Employs browser automation for web scraping, allowing interaction with dynamic web pages and extraction of information.UsageWith CLIycscraperscrape-company--company-urlhttps://www.workatastartup.com/companies/example-incThis command will scrape data for the specified company and save it in the default output format (JSON).With Libraryfromycombinator_scraperimportScraperscraper=Scraper()company_data=scraper.scrape_company_data("https://www.workatastartup.com/companies/example-inc")print(company_data.model_dump_json(by_alias=True,indent=2))Pydantic is used under the hood so methods likemodel_dump_jsonare available for all the scraped data.You can view more examples here:ExamplesContributionWe welcome contributions from the community! To contribute to this project, follow the steps below.Setting Up Development EnvironmentGitpodYou can use Gitpod, a free online VS Code-like environment, to quickly start contributing.Local SetupClone the repository:gitclonehttps://github.com/nneji123/ycombinator-scraper.gitcdycombinator-scraperCreate a virtual environment (optional but recommended):python-mvenvvenvsourcevenv/bin/activate# On Windows: venv\Scripts\activateInstall dependencies:pipinstall-rrequirements.txtRunning TestsMake sure to run tests before submitting a pull request.pipinstall-rrequirements-test.txt
pytesttestsInstalling Documentation RequirementsIf you make changes to documentation, install the necessary dependencies:pipinstall-rrequirements-docs.txt
mkdocsserveSetting Up Pre-Commit HooksWe usepre-committo ensure code quality. Install it by running:pipinstallpre-commit
pre-commitinstallNow,pre-commitwill run automatically before each commit to check for linting and other issues.Submitting a Pull RequestFork the repository and create a new branch for your contribution:gitcheckout-bfeature-or-fix-branchMake your changes and commit them:gitadd.
gitcommit-am"Your meaningful commit message"Push the changes to your fork:gitpushoriginfeature-or-fix-branchOpen a pull request on GitHub. Provide a clear title and description of your changes.DocumentationThedocumentationis made withMaterial for MkDocsand is hosted byGitHub Pages.LicenseYCombinator-Scraper is distributed under the terms of theMITlicense.