in this video i'm going to finish the refactoring of a pdf and web scraping script if you haven't watched part NUMBER_TOKEN of this refactoring i've put a link to that video in the description of this one now i'm not a data scientist myself i'm a software engineer or software designer so i view projects like these through the lens of a software engineer if you want to learn more about data science itself skillshare who's the sponsor of this video has a couple of great classes to get you started skillshare is an online learning community with thousands of inspiring classes for creators explore new skills deepen existing passions and get lost in creativity skillshare has many classes on web development programming in python software engineering and software design at the moment i'm following frank kane's course on data science and machine learning with python it's really comprehensive it contains lessons about statistics data types clustering algorithms decision trees conference libraries such as pandas basically everything you need to know in order to get started skillshare is curated for learning there are no ads and they're always launching new premium classes so you can stay focused and follow wherever your creativity takes you the first NUMBER_TOKEN of my subscribers to click the link in the description will get a one month free trial of skillshare so you can start exploring your creativity today the project i'm refactoring is a script that scrapes data from websites pdf files in particular it looks at academic papers and extracts research keywords and word frequencies the state that we ended up with after the last part is that we have a main file that contains configuration settings i didn't really change anything there we have a couple of classes for handling different scrape requests and what i mentioned last time is that the scrape request class has a very weird mechanism of instantiating sub classes itself which we need to fix so there's a scrape request generic class and then we have scihop scrape we have a json scrape class and we had a pdf scraper which i moved in the last video to a separate file and i cleaned up completely so that pdf scraper now is a pretty short class it has a single method called scrape and it has a couple of helpful methods for guessing the dui computing filter tokens depending on stop words etc etc and a most common words helper function and it uses those functions to produce the scrape result and the result of scrape that's actually a definition we added here it's a data class and it contains a couple of fields now next to these scrapers we have a couple of helper functions here like for changing a directory and there is also a couple of different file request classes which are actually not really all file requests but that basically follows the same structure as the scrape request where the super class is itself creating instances of its own subclasses which is not a good idea like i mentioned in the previous video now the reason these classes are still here is that we still have a dui request class and we have a pop id request class for fetching the data from the pdf files i created a simple function fetch terms from pdf files that solve this and then we no longer need that class so we need to do the same thing for the other two for dui and for the pop id request classes and then after that we can remove this fire request
START_SPONSOR_TOKEN you want to learn more about data science itself skillshare who's the sponsor of this video has a couple of great classes to get you started skillshare is an online learning community with thousands of inspiring classes for creators explore new skills deepen existing passions and get lost in creativity skillshare has many classes on web development programming in python software engineering and software design at the moment i'm following frank kane's course on data science and machine learning with python it's really comprehensive it contains lessons about statistics data types clustering algorithms decision trees conference libraries such as pandas basically everything you need to know in order to get started skillshare is curated for learning there are no ads and they're always launching new premium classes so you can stay focused and follow wherever your creativity takes you the first NUMBER_TOKEN of my subscribers to click the link in the description will get a one month free trial of skillshare so you can start exploring your creativity today the END_SPONSOR_TOKEN