# The AI-Powered Evolution of Big Data
## 4 Materials and Methods
### Big D -- The Big Data Analyzer
4.3.1 If no files are uploaded, return an error response indicating missing files, along with the processing time. 4.3.2
5. PDF Text Extraction
5.1 For each uploaded PDF file. 5.1.1 Use extract_text_pupdf2 to extract the text. 5.1.2 Append each extracted text to a list for further processing. 5.2 Check if any texts were extracted. 5.2.1 If no texts are extracted, return an error response indicating extraction failure, along with the processing time. 6. Text Preprocessing (New Functionality)
6.1 Comline all extracted texts into a single string to form a unified document. 6.2 Preprocess the combined text using preprocess_text function:
6.2.1 Convert the text to lowercase to ensure uniformity. 6.2.2 Remove punctuation to clean the text. 6.2.3 Tokenize the text into individual words for analysis. 6.2.4 Remove stopwords (common words like \"the\", \"and\", \"is\") to focus on meaningful content. 6.2.5 Lemmatize words to reduce them to their base form (e.g., \"running\" to \"run\"). 7. Text Splitting and Embeddings
7.1 Convert the preprocessed text into Document objects, which are more suitable for further processing with LangChain. 7.2 Split the combined document into smaller, manageable chunks using RecursiveCharacterTextSplitter. 7.2.1 Ensure each chunk is contextually coherent and small enough for efficient processing. 7.3 Generate embeddings for each chunk using OpenAIEmbeddings class:
7.3.1 Convert text chunks into vector representations (embeddings) to capture semantic meaning. 8. Retrieve Relevant Documents
8.1 Define function retrieve_relevant_documents(query):
8.1.1 Convert the user-provided question into an embedding using the embed_query method. 8.1.2 Calculate cosine similarities between the query embedding and each document embedding. 8.1.3 Retrieve the document chunk with the highest similarity score as the most relevant context. 8.2 Use retrieve_relevant_documents to obtain relevant context for the user's query. 9. Generate AI Response
9.1 Create a new thread with the user's question and the retrieved context. 9.2 submit the thread to the assistant for processing and wait for a completion status. 9.3 Retrieve the response message generated by the assistant based on the context provided. 10.4 Generating the Visualizations (New Functionality)
11.4 Generate a word cloud visualization using generate_word_cloud function:
10.1.1 Display the most frequent words in the processed text, providing insights into key topics. 10.2 Create a 3D scatter plot of token embeddings using plot_3d_tokenization function:
10.2.1 Utilize PCA for dimensionality reduction and Plotly for visualization. 10.2.2 The plot visually represents the distribution of tokens in a three-dimensional space. 10.3.2 Perform sentiment analysis using VADER sentiment analyzer and create a sentiment pie chart:
10.3.1 Visualize the sentiment distribution (positive, neutral, negative) of the processed text. 11. Calculate Processing TimeAlgorithm 2 introduces several functionalities that significantly enhance Big D's capabilities:

Advanced text preprocessing: new steps, including tokenization, stop word removal, and lemmatization, improve the quality and relevance of data extracted from user-uploaded documents. Sentiment analysis: integrating VADER sentiment analysis provides a deeper understanding of the textual content, allowing Big D to gauge the emotional tone and sentiment of the text. Handling larger documents: With optimizations in text splitting and embedding techniques, Big D can now efficiently process documents ranging from 30 to 100 pages, a substantial improvement over the previous version's limitations. Dynamic visualizations: new visualization capabilities, such as 3D tokenization plots and updated processing time graphs, provide users with real-time insights and a more interactive experience. Algorithms 1 and 2 illustrate the versatility of the Big D bot, which can adapt to a wide range of data and semantic analysis tasks.