Repository: Clueless-Community/scrape-up (language: python)

Issue:
Wikipedia Scraper [GSSoC'23]
**Is your feature request related to a problem? Please describe.**
Yes, the problem is that there is a need for a Wikipedia scraper to extract information from Wikipedia pages automatically. A Wikipedia scraper would be useful for various purposes such as data analysis, research, content aggregation, and more. By automating the process of extracting information from Wikipedia pages, users can save time and effort by retrieving the exact data they need in a structured format. This allows for efficient data collection and analysis, and enables the creation of customized datasets for specific research or application purposes.

**Describe the solution you'd like**
The solution is to create a Wikipedia scraper that can retrieve specific information from Wikipedia pages without manual intervention. Various data extraction operations can be performed using BeautifulSoup. The code will demonstrate examples of extracting the title, introduction paragraph, headings, links, and references from the Wikipedia page. The extracted data will be printed to the console for demonstration purposes. In a real-world scenario, you could store the data in variables, write it to a file, or process it further according to your requirements.

**Describe alternatives you've considered**
Alternative solutions would include manually copying and pasting the desired information from Wikipedia pages or relying on pre-existing datasets. However, these methods can be time-consuming, error-prone, and may not provide the flexibility to retrieve specific and up-to-date information.

I request @nikhil25803 to kindly assign this issue to me under GSSoC'23. Thanks!


Discussion / hints:
Great Idea @prernamittal 
**Note**
+ Create a separate module for this, as per the folder and project structure
+ Work on each method, like create a method first to scrape the detail of a topic. Say Elon Musk for time being
Folder name - `wikipedia`
Class name - `Wikipedia`


Fix patch:
diff --git a/src/scrape_up/wikipedia/__init__.py b/src/scrape_up/wikipedia/__init__.py
new file mode 100644
index 00000000..8c46896f
--- /dev/null
+++ b/src/scrape_up/wikipedia/__init__.py
@@ -0,0 +1,3 @@
+from scrape_up.wikipedia.wikipedia import wikipedia
+
+__all__ = ["wikipedia"]
diff --git a/src/scrape_up/wikipedia/wikipedia.py b/src/scrape_up/wikipedia/wikipedia.py
new file mode 100644
index 00000000..5addaf24
--- /dev/null
+++ b/src/scrape_up/wikipedia/wikipedia.py
@@ -0,0 +1,37 @@
+from bs4 import BeautifulSoup
+import requests
+import json
+
+class WikipediaScraper:
+    def __init__(self, url):
+        self.url = url
+
+    def scrape(self):
+        response = requests.get(self.url)
+        soup = BeautifulSoup(response.text, 'html.parser')
+
+        # Extract the title
+        title = soup.find(id='firstHeading').text
+
+        # Extract all the headings and their content
+        sections = soup.find_all('h2')
+        data = {}
+        for section in sections:
+            heading = section.find('span', class_='mw-headline')
+            if heading:
+                content = []
+                next_node = section.find_next_sibling(['h2', 'h3', 'h4', 'h5', 'h6'])
+                while next_node and next_node.name != 'h2':
+                    if next_node.name in ['h3', 'h4', 'h5', 'h6']:
+                        content.append({'heading': next_node.text.strip()})
+                    elif next_node.name == 'p':
+                        content.append({'text': next_node.text.strip()})
+                    next_node = next_node.find_next_sibling(['h2', 'h3', 'h4', 'h5', 'h6', 'p'])
+                data[heading.text] = content
+
+        # Return the data as JSON
+        result = {
+            'title': title,
+            'sections': data
+        }
+        return json.dumps(result, indent=4)


Test patch:
diff --git a/src/test/wikipedia_test.py b/src/test/wikipedia_test.py
new file mode 100644
index 00000000..fa1e79e8
--- /dev/null
+++ b/src/test/wikipedia_test.py
@@ -0,0 +1,7 @@
+from scrape_up.wikipedia.wikipedia import WikipediaScraper
+
+url = 'https://en.wikipedia.org/wiki/Web_scraping'
+scraper = WikipediaScraper(url)
+scraped_data = scraper.scrape()
+print(scraped_data)
+