# The AI-Powered Evolution of Big Data
## 2 Evolution of Big Data and Revolution in AI  
The rapid and massive rise of artificial intelligence (AI) and the rapid expansion of digital connectivity have ushered in an era dominated by Big Data. The origins of this change go back to the technological infrastructure that developed in the mid-twentieth century, especially during the 1940s. During this period, the rise of relational database management systems (RDBMSs) and the rapid increase in information laid the foundation for a new data-driven world. The advent of groundbreaking developments such as the ENIAC and UNIVAC computers [2] marked the beginning of electronic computing and laid the foundation for the basic technologies that would shape future computing. In the 2000s, the idea of Big Data arose due to the emergence of Web 2.0 [3; 4; 5] and the widespread use of social media such as Facebook (now known as Meta) and Google. The Hadoop distributed file system (HDFS) stores data across large clusters, enabling applications to manage data using a distributed file system. It represents a revolution in distributed file systems and the MapReduce framework within Hadoop enables effective data storage and processing. Apache released Hadoop (Version 3.4.0 ([[https://hadoop.apache.org/](https://hadoop.apache.org/)]([https://hadoop.apache.org/](https://hadoop.apache.org/)))) as open-source software in 2006, after which it was utilized for web indexing by Yahoo and for data handling by Facebook. More importantly, Hadoop has paved the way for the further development of distributed analytics frameworks in the cloud, as well as AI-driven (artificial intelligence) analytics systems [6], which are important for managing and processing large datasets. The term \"Big Data\" was coined in this period [7], reflecting the growing acceptance of the volume and importance of data. Significant progress was made in the 2010s with the advent of the iPhone, which completely transformed the field of mobile technology and enabled the swift expansion of the Internet of Things (IoT) [8]. This period witnessed the extensive use of data science and analytics, as well as the expansion of cloud computing. Collectively, these developments revolutionized the methods of data collection, storage, and analysis. The massive volumes of data generated globally have become central to AI and machine learning. Natural language processing has significantly advanced by integrating transformer models, such as BERT and the GPT family (e.g., GPT-2 and GPT-3) [9; 10]. The broader applications of the transformer architecture make AI tools like ChatGPT into powerful 'Big Calculators' for use in data analysis, even though they are not directly derivatives of BERT or sentence transformers. This development also affected data ethics, privacy concerns, and information needs. The COVID-19 pandemic accelerated the adoption of digital technologies and pushed the boundaries of AI and ML development [11]. During the pandemic, Big Data technologies were adopted by the retail, healthcare, banking, and finance sectors, to name a few. Amazon optimized supply chain logistics using AI, Walmart utilized AI capabilities as well [12; 13], and IT departments in the healthcare sector improved their research methods and then started to create models in order to fast-forward development, as we can see [14; 15; 16]. The banking and finance sectors deploy this for completely digital transactions, leading to real-time risk avoidance [17]. Some well-known examples include Deutsche Bank, BBVA Fujitsu, and Hokuhoku Financial Group [17]. The COVID-19 pandemic accelerated the digitization of many industries, leading to rapid growth in the tech/IT sector, including in the area of AI. The rise of AI during pandemics is detailed in Appen's 2020 State of AI and Machine Learning Report, which can be found online [18; 19].