Package: lrabbit-scrapy
lrabbit_scrapythis is a small spider,you can easy running. When you often need to crawl a single site, you don't have to redo some
repeated code every time, using this small framework you can quickly crawl data into a file or database.Installing$ pip install lrabbit_scrapyquick startcreate blog_spider.pypython3 -m lrabbit_scrapy new_scrapy blogthis is file contentfromlrabbit_scrapy.spiderimportLrabbitSpiderfromlrabbit_scrapy.common_utils.network_helperimportRequestSessionfromlrabbit_scrapy.common_utils.print_log_helperimportLogUtilsfromlrabbit_scrapy.common_utils.all_in_oneimportFileStoreimportosfromlrabbit_scrapy.common_utils.mysql_helperimportMysqlClientfromparselimportSelectorclassSpider(LrabbitSpider):"""spider_name : lrabbit blog spider"""# unique spider namespider_name="lrabbit_blog"# max thread worker numbersmax_thread_num=2# is open for every thread a mysql connection,if your max_thread_num overpass 10 and  in code need mysql query ,you need open this configthread_mysql_open=True# reset all task_list,every restart program will init task listreset_task_config=False# open loop init_task_list ,when your task is all finish,and you want again ,you can open itloop_task_config=False# remove config option,if open it,then confirm option when you init taskremove_confirm_config=False# config_path_name, this is env name ,is this code ,you need in linux to execute: export config_path="crawl.ini"config_env_name="config_path"# redis db_numredis_db_config=0# debug log ,open tracback logdebug_config=Falsedef__init__(self):super().__init__()self.session=RequestSession()self.proxy_session=RequestSession(proxies=None)csv_path=os.path.join(os.path.abspath(os.getcwd()),f"{self.spider_name}.csv")self.field_names=['id','title','datetime']self.blog_file=FileStore(file_path=csv_path,filed_name=self.field_names)defworker(self,*args):task=args[0]mysql_client:MysqlClientiflen(args)==2:mysql_client=args[1]# mysql_client.execute("")res=self.session.send_request(method='GET',url=f'http://www.lrabbit.life/post_detail/?id={task}')selector=Selector(res.text)title=selector.css(".detail-title h1::text").get()datetime=selector.css(".detail-info span::text").get()iftitle:post_data={"id":task,"title":title,'datetime':datetime}self.blog_file.write(post_data)# when you succes get content update redis statself.update_stat_redis()LogUtils.log_finish(task)definit_task_list(self):# you can get init task from mysql# res = self.mysql_client.query("select id from rookie limit 100 ")# return [task['id'] for task in res]return[iforiinrange(100)]if__name__=='__main__':spider=Spider()spider.run()set config.ini and config env variablecreate crawl.ini,for example this file path is /root/crawl.ini[server]mysql_user=rootmysql_password=123456mysql_database=testmysql_host=192.168.1.1redis_user=lrabbitredis_host=192.168.1.1redis_port=6379redis_password=123456[test]mysql_user=rootmysql_password=123456mysql_database=testmysql_host=192.168.1.1redis_user=lrabbitredis_host=192.168.1.1redis_port=6379redis_password=123456set config envwindows power shell$env:config_path = "/root/crawl.ini"linuxexport config_path="/root/crawl.ini"python3 blog_spider.pyother functionpython3 blog_spider.py statshow task statpython3 -m lrabbit-scrapy sslpasspass android ssl
Linksauthor:https://www.lrabbit.life/