Package: profilescout
Profile ScoutTable of ContentsAboutCapabilitiesCommon Use CasesScrapingProfile related tasksInformation extractionInstallationPyPISourceHostDocker containerPossibilities for future improvementsContributingAboutProfile Scoutis a versatile Python package that offers scraping and detection capabilities for profile pages on any given website, including support for information extraction. By leveraging its robust search functionality and machine learning, this tool crawls the provided URL and identifies the URLs of profile pages within the website. Profile Scout offers a convenient solution for extracting user profiles, gathering valuable information, and performing targeted actions on profile pages. With its streamlined approach, this tool simplifies the process of locating and accessing profile pages, making it an invaluable asset for data collection, web scraping, and analysis tasks. Additionally, it supports information extraction techniques, allowing users to extract specific data from profile pages efficiently.Profiel Scout can be useful to:Investigators andOSINT Specialists(information extraction, creating information graphs, ...)Penetration Testersand Ethical Hackers/Social Engineers(information extraction, reconnaissance, profile building)Scientists and researchers (data engineering, data science, social science, research)Companies (talent research, marketing, contact acquisition/harvesting)Organizations (contact acquisition/harvesting, data collecting, database updating)CapabilitiesProfile Scout is mainly a crawler. For given URL, it will crawl the site and perform selected actions.
If the file with URLs is provided, each URL will be processed in seperate thread.Main features:Flexible and controlled page scraping (HTML, page screenshot, or both)Detecting and scraping profile pages during the crawling processLocating the collective page from which all profile pages originate.Information extraction from HTML filesOptions:-h, --help            
    show this help message and exit
    
--url URL             
    URL of the website to crawl
    
-f URLS_FILE_PATH, --file URLS_FILE_PATH
    Path to the file with URLs of the websites to crawl
    
-D DIRECTORY, --directory DIRECTORY
    Extract data from HTML files in the directory. To avoid saving output, set '-ep'/'--export-path' to ''

-v, --version
    print current version of the program

-a {scrape_pages,scrape_profiles,find_origin}, --action {scrape_pages,scrape_profiles,find_origin}
    Action to perform at a time of visiting the page (default: scrape_pages)
    
-b, --buffer          
    Buffer errors and outputs until crawling of website is finished and then create logs
    
-br, --bump-relevant  
    Bump relevant links to the top of the visiting queue (based on RELEVANT_WORDS list)
    
-ep EXPORT_PATH, --export-path EXPORT_PATH
    Path to destination directory for exporting
    
-ic {scooby}, --image-classifier {scooby}
    Image classifier to be used for identifying profile pages (default: scooby)
    
-cs CRAWL_SLEEP, --crawl-sleep CRAWL_SLEEP
    Time to sleep between each page visit (default: 2)
    
-d DEPTH, --depth DEPTH
    Maximum crawl depth (default: 2)
    
-if, --include-fragment
    Consider links with URI Fragment (e.g. http://example.com/some#fragment) as seperate page
    
-ol OUT_LOG_PATH, --output-log-path OUT_LOG_PATH
    Path to output log file. Ignored if '-f'/'--file' is used
    
-el ERR_LOG_PATH, --error-log-path ERR_LOG_PATH
    Path to error log file. Ignored if '-f'/'--file' is used
    
-so {all,html,screenshot}, --scrape-option {all,html,screenshot}
    Data to be scraped (default: all)
                
-t MAX_THREADS, --threads MAX_THREADS
    Maximum number of threads to use if '-f'/'--file' is provided (default: 4)
    
-mp MAX_PAGES, --max-pages MAX_PAGES
    Maximum number of pages to scrape and page is considered scraped if the action is performed successfully (default: unlimited)
    
-p, --preserve        
    Preserve whole URI (e.g. 'http://example.com/something/' instead of 'http://example.com/')

-r RESOLUTION, --resolution RESOLUTION
    Resolution of headless browser and output images. Format: WIDTHxHIGHT (default: 2880x1620)

Full input line format is: '[DEPTH [CRAWL_SLEEP]] URL"

DEPTH and CRAWL_SLEEP are optional and if a number is present it will be consider as DEPTH.
For example, "3 https://example.com" means that the URL should be crawled to a depth of 3.

If some of the fields (DEPTH or CRAWL_SLEEP) are present in the line then corresponding argument is ignored.

Writing too much on the storage drive can reduce its lifespan. To mitigate this issue, if there are more than
30 links, informational and error messages will be buffered and written at the end of
the crawling process.

RELEVANT_WORDS=['profile', 'user', 'users', 'about-us', 'team', 'employees', 'staff', 'professor', 
                'profil', 'o-nama', 'zaposlen', 'nastavnik', 'nastavnici', 'saradnici', 'profesor', 'osoblje', 
                'запослен', 'наставник', 'наставници', 'сарадници', 'професор', 'особље']Common Use CasesNote: Order of arguments/switches doesn't matterScrapingScrape the URL up to a depth of 2 (-d) or a maximum of 300 scraped pages (-mp),
depending on which comes first. Store scraped data at/data(-ep)profilescout--urlhttps://example.com-d2-mp300-ep/dataScrape HTML (-so html) for every page up to a depth of 2 for the list of URLs (-f).
Number of threads to be used is set with-tprofilescout-ep/data-t`nproc`-flinks.txt-d2-sohtmlStart scraping screenshots from specific page (-p). It is import to note here that
without-p, program would ignore full path, to be precise/about-us/meet-the-team/partprofilescout-p--urlhttps://www.wowt.com/about-us/meet-the-team/-mp4-soscreenshotScrape each website in the URLs list and postpone writing to the storge disk (by using buffer,-b)profilescout-b-t`nproc`-flinks.txt-d0-ep/dataProfile related tasksScrape profile pages (-a scrape_profiles) and prioritize links that are relevant to some specific domain (-br).
For example, if we were searching for profile pages of professors we would like to give priority to links that
contain related terms which could lead us to the profile page. Note: you can change it in fileconstants.pyprofilescout-br-t`nproc`-flinks.txt-ascrape_profiles-mp30Find and screenshot profile, store it as 600x400 (-r) image and then wait (-cs) 30 seconds before moving to the next profileprofilescout-br-t`nproc`-flinks.txt-ascrape_profiles-mp1000-d3-cs30-r600x400Locate the origin page of profile pages (-a locate_origin) with classifier calledscooby(-ic scooby).
Note that visited pages are lond so in can be used for something like scanning the websiteprofilescout-t`nproc`-flinks.txt-alocate_origin-icscoobyInformation extractionExtract information (-D) contained in profile HTMLs that are located at/dataand store it at~/results(-ep)profilescout-D/data-ep~/resultsInstallationPyPipip3installprofilescoutSourceHostCreate virtual environment (optional, but recommended)python3-mvenv/path/to/some/dirActivate virtual environment (skip if you skipped the first step)source/path/to/some/dir/bin/activateInstall requirementspip3install-rrequirements.txtInstall package locallypip3install-e.Exploreprofilescout-hDocker containerCreate image and run container. Execute this in project's directorymkdir"/path/to/screenshot/dir/"# if it does not exist# this line may differ depending on your shell,# so check the documentation for the equivalent file to .bashrcecho'export SS_EXPORT_PATH="/path/to/screenshot/dir/"'>>~/.bashrc
dockerbuild-tprofilescout.
dockerrun-it-v"$SS_EXPORT_PATH":/dataprofilescoutAdd--rmif you want it to be disposable (one-time task)Test deployment (inside docker container)profilescout-mp4-t1-ep'/data'-p--urlhttps://en.wikipedia.org/wiki/GNUPossibilities for future improvementsClassificationProfile classification based on existing data (without crawling)Classification using HTML and images, as well as the selection of appropriate classifiersScrapingIntelligent downloading of files through links available on the profile pageCrawlingSupport for scraping using proxies.Crawling actionsAbility to provide custom actionsActions before and after page loading.Multiple actions for each stage of page processing (before, during, and after access).Crawling strategyAbility to provide custom heuristicsAbility to choose crawling strategy (link filters, etc.)Support for deeper link bumpSelection of relevant words using CLIUsabilitySaving progress and the ability to resumeIncreased automation (if the profile is not found at depth DEPTH, increase the depth and continue).ExtractionSupport for national numbers, e.g.011/123-4567Experiment with lightweight LLMsExperiment with Key-Value extraction and Layout techniques likeLayoutLMContributingIf you discover a bug or have a feature idea, feel free to open an issue or PR.Any improvements or suggestions are welcome!