Big Data vs Business Intelligence
By: Alex Mosso
Tags: Big Data Analytics, Software Engineering, Big Data, Software Development, Business Intelligence

In many articles we can find a definition of big data. But now let’s put big data in perspective. Before big data, what we used to do was implement business intelligence models. In short, business intelligence allows us to create conceptual models which help us to analyze historical and current data, and also it might create statistical models to predict future. On the other hand, big data doesn’t create conceptual models only, but focuses on creation of mathematical models, and it’s able to predict future behavior better, answer known and unknown questions, and take decisions on real time.

In business intelligence, analysts use queries to extract information from database. In contrast, big data proposes to use sophisticated data manipulation techniques, such as —data extraction, data partitioning, parallelism, concurrence, optimization — And takes a variety of sources, not only databases (structured data), but also unstructured data (images, voice, video, documents), and semi-structured data (delimited files, logs, NoSQL databases).

Another difference between business intelligence and big data is the volume of information we process under each discipline.

From all these statements, we can deduce the V’s of big data —Volume, Velocity, Variety, Visualization —and the most important one — Value.

The roles in each discipline

Business intelligence analysts develop very optimized queries to extract information, while in big data, a data engineer develops sophisticated and optimized processes to extract information.

Whereas in business intelligence an analyst designs conceptual models and elaborates reports, in big data, a data scientist uses advanced analytical, probabilistic and statistical models, along as AI algorithms like Neural Networks or Genetic Algorithms.

Finally, a business intelligence analysts must have some knowledge regarding business to present relevant strategic dashboards and tactical KPI’s. But in big data, we have subject matter experts who interpret reports and graphics, and give feedback to data engineers and data scientists.

The impact in business

Big data allows us to transform data into knowledge, and whenever possible let the applications take real time decisions; whereas the purpose of business intelligence is to create a base line of reports to let operational workers take better decisions.

Big data let us put data in the middle of corporate business strategy, generating what we call data lakes, which enables all organization to have access to data. And if we have to think in the V for Value, we need to see where is that value in big data, the return of investment.

Any use case we could imagine for big data, for any industry, and for any sector, it is resumed on three major axes:

Big data let us generate new income sources thanks to personalization. It let us know better our customers, micro-segment them based on their behavior, and develop new products and services much more personalized; and therefore, it allows us to generate new income sources, recommendation engines, much more segmented offers. Big data enables us to enhance decision making and improve operational efficiency. Decision making is something very important for organizations, and so far, those decision has been taken with help of tools like business intelligence, with the help of reports and dashboards where we can see performance and activity in the company, and based on that we take decisions for the future. But business intelligence nowadays is quite limited. From a business point of view, big data has more powerful tools and methods to predict and for decision making (such as machine learning or deep learning), therefore, big data enables us to take better decisions in a the company. Finally, big data enables us to predict and prevent fraud and risk, such as failures prediction, customer loss prediction, bad debt prediction, delinquency prediction.

Therefore, any use case for any business is covered by these three axes.

How to do it the big data way

This methodology has four high level components: Business, Technology, Data science, and Communication.

This methodology is composed of eight phases.

Business understanding: We want to know what is the problem and what is the objective. Know your data: Here we identify data sources at high level, but also we do data profiling to identify data patterns and detailed characteristics. Data profiling is done with help of data science team. We also are expecting to have static and dynamic models of our data. Technology stack: We define technology, products, frameworks, hardware requirements, networking requirements. Data processing: Here we design data pipelines, and apply data processing techniques, such as — data partitioning, filtering, data transformation, joining, sorting, normalization, aggregation, parallelism. Modeling: Here we create models that allows us to extract knowledge from data. For example, if we are using neural networks, our model will be purely mathematical; it means that for each input data set, data scientist will need to find numerical matrix representation along with polynomial functions to process matrix (this is what we know as deep learning). Results presentation: Whatever we have found, we need to present it and communicate it to our stake-holders with help of visualization tools. Deployment: Once we have approval from subject matter experts, we deploy our model in production. Value adding: Once we have the model in production, subject matter experts will start mining the model.

In the past, a technology stack required to process large amounts of data was quite costly, and only big corporations could deploy solutions to process large amounts of data, although without the data science component. Those technologies are Mainframe, Cobol, Tandem, Ab Initio; undoubtedly, these technologies are robust, and able to process really huge amounts of data, but expensive.

In the past, in order to do a data profiling we required highly specialized tools (and not within the reach of anyone), but now we can do it with free tools like R.

Right now, big data is already changing industries, and the way of doing business.