Position: Web scraper (Python or Node, AWS)
Company: Codemotion
Category: Other
Experience required: 5y
English level: upper
Published: 2023-06-01T00:00:00+03:00

Level: Senior

Hard skills requirements (including years):
1- Proven experience in web scraping, with a track record of successfully extracting data from various sources, including cryptocurrency exchange website
2- Proficiency in programming languages commonly used for web scraping, such as Python or Node.js. (min 4-5 years)
3-Strong understanding of scraping techniques, frameworks, and libraries, with a focus on achieving high-speed scraping.
4- Knowledge of proxy management to handle IP limitations and ensure uninterrupted scraping operations.
5- Extensive experience with AWS infrastructure and services, including EC2, S3, Lambda, and CloudWatch, to build scalable and reliable systems.

Project tasks:
Designing and implementing high-speed and stable scraping solutions for both exchanges and Twitter, leveraging your expertise in AWS infrastructure, proxies, IP limitations, and Websockets;
Detailed scope of the work.

Overview:
Develop a high-speed web scraper capable of extracting announcements from the Binance and Upbit cryptocurrency exchanges. The scraped data will be utilized for trading purposes and require efficient delivery to a trading terminal.

Data Extraction:
a. Access and scrape the announcement sections of Binance and Upbit platforms.
b. Extract key information such as title, content, date, and relevant metadata from the announcements.
c. Support retrieval of both historical data and real-time updates.

Integration with Trading Terminal:
a. Establish a method to transmit the scraped data to a trading terminal.
b. Ensure smooth and timely delivery of data updates.
c. Explore options like WebSocket for efficient data transmission.

Competitor Benchmarking:
a. Conduct analysis of existing competitors and their web scraping capabilities.
b. Set performance benchmarks based on competitor speeds and WebSocket efficiency.
c. Strive to outperform competitors and provide faster data retrieval and WebSocket performance.

Telegram/Twitter Scraping Integration:
a. Implement scraping capabilities for Telegram/Twitter data.
b. Enable HTTP scraping or integration with libraries like Telethon for Telegram data retrieval.
c. Retrieve relevant information from Telegram channels or groups.

Performance Optimization:
a. Implement efficient data retrieval algorithms to enhance scraping speed.
b. Explore techniques like asynchronous processing and parallelization for improved performance.
c. Continuously monitor and optimize the system to maintain competitiveness.

English level: Upper-Intermediate

Planned duration: 4-5 months full-time, then will depend on task

Team geography: Turkey
Current methodology: Agile