Package: instacrawl
InstaCrawlAnInstagrampost crawler focused on optimizing algorithm performance through model classificationInstallationWith pippipinstallinstacrawlFrom sourcegitclonehttps://github.com/annie444/instacrawl.gitcdinstacrawl

pipinstall-rrequirements.txt

pythoninstacrawl/__main__.pyNOTE:InstaCrawl requiresGoogle Chromeandchromedriverto be installed. ThePyPipackage comes with an installation ofchromedriver, however, there is no guarantee that this version ofchromedriverwill be compatible with your version ofGoogle Chrome.Using InstaCrawlSetupTo launch InstaCrawl, open a new terminal window and typeinstacrawl.You'll end up at a selection dialogue to select where you'd like to start. The reason for this is that InstaCrawl can take a while to get through all the steps in some cases. This way you can stop after each step and resume the analysis later.╭─────────────────────────────────InstaCrawl─────────────────────────────────╮
│Whatwouldyouliketodo?│
││
││
││
││
││
││
││
││
││
│Press1tostartcrawlinganewprofile│
│Press2toscrapesaveddatafromacrawl│
│Press3todownloadpostsfromacrawl│
│Press4toalignacrawl│
│Press5toanalyzeacrawl│
│Press6toviewtheresultsfromacrawl│
│Press0toexit│
││
╰──────────────────────────────────────────────────────────────────────────────╯Step 1.After selecting step one you'll be prompted to enter the account you wish to analyze alongside the credentials to the instagram account you'd like to use for accessing the account. Your login info will be saved in a~/.instacrawlrcfile for future reference. You then will be prompted for a directory to save the data to as well as a date you'd like to crawl back to. You can leave the date empty to just crawl the first ~21 posts on the account.During step 1Google Chromewill open repeatedly and it will say that the site is unsafe. (the HTTPS/security chip on the URL bar will be red.) This is expected. At this point, InstaCrawl is using a proxy to intercept all data between your computer andInstagramto parse out your posts. Facebook uses a clever technique of changing the way they send posts from their servers to your computer. To get around this, InstaCrawl may either:take longer than usual to parse out a post, which very well may look like it's stalled. Do not worry, InstaCrawl tries many different parsing methods to sort through the garbage and find the necessary information.Google Chromemay open and close a few times. This is usually the result ofInstagramcatching onto the crawler and trying to block InstaCrawl's attempts to scrape data. In order to circumvent this, InstaCrawl will do multiple rounds of deleting cookies and changing proxies.At the end of step 1, InstaCrawl will save all of the scraped data in astep_one.pklfile. There are programs that can open these types of files, but all that's been collected thus far is:The post codesThe number of likesThe number of commentsThe basic post metadataStep 2.The point of point two is to scrape the actual media from the posts. Or at least to prepare to do that. This step in particular often upsets the servers, so during this step InstaCrawl will most likely repeatedly refresh it's cookies. InstaCrawl will try to do as much of step 2 in the background, but it's likely thatGoogle Chromewill open at least once during this step.After step 2, InstaCrawl will save the collected post data in astep_two.pklfile. At this point all of the publicly available data has been collected. In addition to step 1, step 2 has collected:The hashed download links to post media (i.e. images and videos)The alt text and accessibility information tied to each postThe page HTML and server JSON for that postThe type of post (e.g. carousel, reel, etc.)Step 3.From here on out, InstaCrawl will mostly work in the background. During step 3 InstaCrawl will open as many threads as possible to download post content in parallel. You'll notice at this point that the folder with all your save data will have numbered folders containing the high quality content from each post in either.jpgor.mp4format. Each folder number lines up with the indices in the data.Step 4.During step 4 you'll get to look at the posts in the terminal (it's not the best, but come on, I got an image to print out on any tty output.) To be honest, this is mostly a gimmick and you can pressSPACEto continue. Next you'll be prompted for a save path of the data thus far. This can be in any format supported byPandas. At this point you can stop if you just wanted the data from the posts, but if you are here for algorithm analytics, keep moving forward.Step 5.Step 5 is the most computationally intense and it may make your computer very slow while it's running. Because of this, I advise that you use something likethis appto keep your computer awake, or at the very least, just plug in your computer and set it to not fall asleep.At the beginning of step 5 you'll probably get a lot of weird errors and warnings. Don't worry, nothing is broken (I hope), InstaCrawl is just initializingdlibandhuggingfacefor analysing your posts against facebook's post analysis models. This part generally looks like it's doing a lot, but it really is just downloading models and packages from the web. You may also notice your RAM being eaten up, but generally little CPU usage. This is InstaCrawl preprocessing all of the posts. Don't worry though, everything will hop out of RAM and into some tempfiles soon.After everything has downloaded and initialized, InstaCrawl will begin running your posts through a slew of machine learning models. Each model's output will be stored in a human-readable format of your choosing (much like it step 4), but there will inevitably beA LOTof data to store. So be prepared, the final output of a business profile's analysis is usually around 3GB.Once complete you'll be prompted for a path and file to save the end data in. This, again, can be in any format supported byPandasand will contain all of the data from step. At this point, this is the completed data.Step 6.Step 6 does nothing itself other than launch an instance ofD-talewith all of the fancy add-ons for you to explore, analyse, and visualize the data. Feel free to come back to step 6 as often as you'd like. Also, note that InstaCrawl comes with all of the necessary packages to export graphs, run statistical analyses, and parse/sort the data. No need to install any additional software/packages.ReferenceIf you want to build of InstaCrawl, you're welcome to! I built it using common OO schemes (although OO isn't my favorite). However, I will warn you that isn't a bit of a nightmare in terms of complex datatypes. But for a weekend project, it is what it is.InstaCrawl is built upon three main classes. The first being the baseInstaCrawlclass in theprompts.pyfile. This class simply handles the control flow of the application and generally isn't that interesting in terms of development, so I've chosen to omit it's docs. If you really want the docs, feel free to open an issue./insta.pyclassCookies(Dict):"""Class to store cookies.This class inherits the Dict class from the typing module,and provides a few helpful abstractions for refreshing thecookies and handling parallel threads attached to a sharedinstance. Convenient when maintaining a singleton pattern....Attributes----------cookies : dict[str, Optional[str]]This is a simple dictionary of cookies with the`cookie_name`: `cookie_value` patterncount : int, optional, default: 0This is used to maintain a count of how often thecookies have been refreshed. This is useful if you'retrying to limit the amount of calls to instagram.comMethods-------refresh(link : str, optional, default = "https://instagram.com") -> Nonerefresh_count() -> int"""defrefresh(self,link:str="https://instagram.com"):"""A method to refresh the cookies with Selenium....Args----link : str, optional, default = "https://instagram.com"Returns-------None"""defrefresh_count(self):"""Return number of times cookies have been refreshed....Returns-------int : the number of times the cookies have been refreshed"""classInstagramData:"""Class to store and scrape data from Instagram profile page.This class does all the heavy lifting when it comes to web scraping....Attributes----------save_to : PathLikeThis is the path where the saved data is stored.Properties----------step_one : InstaStepOnestep_two : InstaStepTwostep_three : InstaStepThreestep_four : InstaStepFourdownload_requests : List[DownloadRequests]Methods-------new(profile_url : str,save_path : Optional[PathLike],username : str,password : str) -> Noneopen() -> Noneclose() -> Nonelogin() -> Noneget_login_data() -> intget_posts( i : int ) -> Noneload_more() -> Noneintercept_posts() -> intget_contents() -> Noneonce_complete(f : concurrent.futures.Future) -> Nonedownload() -> Noneonce_downloaded(f : concurrent.futures.Future) -> Nonemore() -> Nonestart() -> Nonerun() -> Nonerun_until( date : datetime.datetime ) -> Noneanalyze() -> Nonepost_link( href : str ) -> boolgenerate( multi : bool ) -> pandas.DataFrameget_cookies() -> Cookiessave_step_one() -> Nonestep_one_saved() -> boolload_step_one() -> None... same methods exist for step_two through step_four"""defopen(self)->None:"""Opens Selenium, ChromeDriver, and a requests session"""defclose(self)->None:"""Closes the Selenium and requests sessionsand wipes them from memory"""deflogin(self)->None:"""Runs the login workflow with Selenium"""defget_login_data(self)->int:"""Intercepts the login data from the server....Returns-------intThe key from the self.posts dictionary for the firstpost processed."""defget_posts(self,i:int)->None:"""Scrape the HTML for post data starting at post `i'"""defload_more(self)->None:"""Load more posts"""defintercept_posts(self)->int:"""Intercept posts as they are sent from the server....Returns-------intThe key from the self.posts dictionary for the firstpost processed."""defget_contents(self)->None:"""Get the post contents from each individual posts' page.NOTE: Runs in parallel threads"""defonce_complete(self,f:concurrent.futures.Future)->None:"""Process the data returned from `get_contents()'"""defdownload(self)->None:"""Download each posts' mediaNOTE: Runs in parallel"""defonce_downloaded(self,f:concurrent.futures.Future)->None:"""Process the data returned from `download()'"""defmore(self)->None:"""Compounded method to load more posts and scrape their data"""defstart(self)->None:"""Compound method that:- Logs into Instagram- Collects the first round of data"""defrun(self)->None:"""Auto grabs the first ~21 posts"""defrun_until(date:datetime.datetime)->None"""Crawls posts until it hits a given date.Args----data : datetime.datetimeWhen to crawl until (e.g. 2021-04-04)"""defanalyze(self)->None:"""This is just an alias for `get_contents()'"""defpost_link(self,href:str)->bool:"""This is a utility method to filter the `href'attributes of anchor tags with BeautifulSoup"""defgenerate(self,multi:bool)->pandas.DataFrame:"""Generates a pandas.DataFrame from the dataArgs----multi : boolWhether or not you would like the resulting DataFrameto have a multi-index on the columnsReturns-------pd.DataFrame"""defget_cookies(self)->Cookies:"""Initializes the Cookies class from the cookies storedin the selenium and requests sessions"""defsave_step_one(self)->None:"""Writes to step_one.pkl"""defstep_one_saved(self)->bool:"""Alias for os.path.exists"""defload_step_one(self)->None:"""Loads all the saved data from the previous step"""/analysis.pyclassPostalyzer:"""The analysis control class for running multithreaded inferenceAttributes----------data : InstaStepFourThe data from step 4Methods-------analyze() -> List[pandas.DataFrame]"""defanalyze(self)->List[pd.DataFrame]:"""Runs and controls the analysis of the dataNOTE: Runs on multiple threadsReturns-------List[pandas.DataFrame]A list of the results from each class of analysis"""classInstaAnalysis(object):"""A base class for each subsequent analysis pipelineAttributes----------data : InstaStepFourThe data from step 4name : strThe name of the analysis target (e.g. 'images')Methods-------insert_result(post: int,cat: str,label: str,data: Any,append: bool = False) -> Noneanalyze() -> Nonesave() -> pandas.DataFrame"""definsert_result(self,post:int,cat:str,label:str,data:Any,append:bool=False)->None:"""Helper method for managing results"""defanalyze(self)->None:"""The actual ML algorithm"""defsave(self)->pd.DataFrame:"""Saves the results to a pandas.DataFrameReturns-------pandas.DataFrameThe results from the ML algorithm"""# Not shown here is three helper methods for displaying the progress# to the terminal.classVideoAnalysis(InstaAnalysis):"""Runs analysis of videos utilizing theMCG-NJU/videomae-base-finetuned-kineticsmodel"""classImageAnalysis(InstaAnalysis):"""Runs analysis on images utilizing the following models:- facebook/detr-resnet-50- mattmdjaga/segformer_b2_clothes- face_recognition_models"""classTextAnalysis(InstaAnalysis):"""Runs analysis on the post's text utilizing:- SamLowe/roberta-base-go_emotions- alimazhar-110/website_classification- cardiffnlp/twitter-roberta-base-sentiment-latest"""