Package: vk-url-scraper
vk-url-scraperPython library to scrape data, and especially media links like videos and photos, from vk.com URLs.You can use it via thecommand lineor as apython library, check thedocumentation.InstallationYou can install the most recent release frompypiviapip install vk-url-scraper.To use the library you will need a valid username/password combination for vk.com.Command line usage# run this to learn more about the parametersvk_url_scraper--help# scrape a URL and get the JSON result in the consolevk_url_scraper--username"username here"--password"password here"--urlshttps://vk.com/wall12345_6789# ORvk_url_scraper-u"username here"-p"password here"--urlshttps://vk.com/wall12345_6789# you can also have multiple urlsvk_url_scraper-u"username here"-p"password here"--urlshttps://vk.com/wall12345_6789https://vk.com/photo-12345_6789https://vk.com/video12345_6789# you can pass a token as well to avoid always authenticating# and possibly getting captcha prompts# you can fetch the token from the vk_config.v2.json file generated under by searching for "access_token"vk_url_scraper-u"username"-p"password"-t"vktoken goes here"--urlshttps://vk.com/wall12345_6789# save the JSON output into a filevk_url_scraper-u"username here"-p"password here"--urlshttps://vk.com/wall12345_6789>output.json# download any photos or videos found in these URLS# this will use or create an output/ folder and dump the files therevk_url_scraper-u"username here"-p"password here"--download--urlshttps://vk.com/wall12345_6789# orvk_url_scraper-u"username here"-p"password here"-d--urlshttps://vk.com/wall12345_6789Python library usagefromvk_url_scraperimportVkScrapervks=VkScraper("username","password")# scrape any "photo" URLres=vks.scrape("https://vk.com/photo1_278184324?rev=1")# scrape any "wall" URLres=vks.scrape("https://vk.com/wall-1_398461")# scrape any "video" URLres=vks.scrape("https://vk.com/video-6596301_145810025")print(res[0]["text"])# eg: -> to get the text from code# Every scrape* function returns a list of dict like{"id":"wall_id","text":"text in this post","datetime":utcdatetimeofpost,"attachments":{# if photo, video, link exists"photo":[listofurlswithmaxquality],"video":[listofurlswithmaxquality],"link":[listofurlswithmaxquality],},"payload":"original JSON response converted to dict which you can parse for more data}see [docs] for all available functions.TODOscrape album linksscrape profile linksdocs online from sphinxDevelopment(more info inCONTRIBUTING.md).setup dev environment withpip install -r dev-requirements.txtorpipenv install -r dev-requirements.txtsetup environment withpip install -r requirements.txtorpipenv install -r requirements.txtTo run all checks tomake run-checks(fixes style) or individuallyTo fix style:black .andisort .->flake8 .to validate lintTo do type checking:mypy .To test:pytest .(pytest -v --color=yes --doctest-modules tests/ vk_url_scraper/to use verbose, colors, and test docstring examples)make docsto generate shpynx docs -> editconfig.pyif neededTo test the command line interface available inmain.pyyou need to pass the-moption to python like so:python -m vk_url_scraper -u "" -p "" --urls ...Releasing new versioneditversion.pywith proper versioningrun./scripts/release.shto create a tag and push, alternativelygit tag vx.y.zto tag versiongit push origin vx.y.z-> this will trigger workflow and put project onpypigo tohttps://readthedocs.org/to deploy new docs version (if webhook is not setup)Fixing a failed releaseIf for some reason the GitHub Actions release workflow failed with an error that needs to be fixed, you'll have to delete both the tag and corresponding release from GitHub. After you've pushed a fix, delete the tag from your local clone withgittag-l|xargsgittag-d&&gitfetch-tThen repeat the steps above.