Big Data: Its Benefits, Challenges, and Future | by Benedict Neo | Towards Data Science

Full seriesPart 1 - What is Data Science, Big data and the Data Science processPart 2 - The origin of R, why use R, R vs Python and resources to learnPart 3 - Version Control, Git & GitHub and best practices for sharing code.Part 4 - The 6 types of data analysisPart 5 - The ability to design experiments to answer your Ds questionsPart 6 - P-value & P-hackingPart 7 - Big Data, it's benefits, challenges, and future
This series is based on the Data Science Specialization offered by John Hopkins University on Coursera. The articles in this series are notes based on the course, with additional research and topics for my own learning purposes. For the first course, Data Scientist Toolbox, the notes will be separated into 7 parts. Notes on the series can also be found here.
Before the internet, information was in some ways restricted and more centralized. The only mediums of information were books, newspapers, and word of mouth, etc. But now with the advent of the internet and improvements to computer technology (Moore’s Law), information and data skyrocketed, and it has become this open-system, where information can be distributed to people without any kind of limits. As the internet became more accessible and world-wide, social mobile applications and websites gradually grew to become platforms for sharing data. Data, along with many other things, grows in value as an increase in size, where this value is applied in many ways, but mostly for analytics and making decisions. Here’s more about Big Data.
Big Data can be defined as large amounts of data, both structured and unstructured, usually stored in the cloud or in data centers, which are then utilized by companies, organizations, startups, and even the government for different purposes.
To utilize data means cleaning it and then analyzing it, forming patterns and connection, trends and correlations, to produce insights. This is what’s called Big Data analytics.
Big Data is also commonly described by its qualities, also known as the 4Vs
Insurmountable amounts of data due to improvements to technology and data storage (cloud storages, better processes, etc)
Data is generated at astonishing rates, related to computer’s speed and capability increasing (Moore’s Law)
Wide range of data of different formats and types easily collected, in an era of social media and the internet.
inconsistencies and uncertainty of data (unstructured data — images, social media, video, etc.)
A brief explainer on structured and unstructured data
Traditional data — tables, spreadsheets, databases with columns and rows, CSV and Excel, etc
rarely how data is today — much messier
job is to extract information and corral it to something tidy and structured
The proliferation of data from digital interactions — email, social media, text, customer habits, smartphones, GPS, websites, activity, video, facial rec,
Big data — new tools and approaches to utilize new data & cleaning and analysis on unstructured data
There are a few popular tools which are commonly associated with big data analytics,
Hadoop
Apache Spark
Apache Hive
SAS
Most of these tools are just open-source frameworks for handling huge data efficiently and helpful features to do so.
R
Python
Scala
These languages are very popular in the data science world and can be used for handling large amounts of data through specific libraries and packages.
Big Data can be seen in many places today. One prevalent example is online retailers. Companies like Amazon are centered on building accurate recommender systems that tailer to their customers, the better the system, the more products their customers would be interested in, which then translates to more sales.
To do this, Amazon would need tons of data, information like purchasing behaviors, browsing and cart history, demographics, etc. Recommender systems that build a profile of users are also seen in social media, streaming services, and many more.
Big Data is also applied in many sectors — Healthcare, Manufacturing, Public sector, media & entertainment, etc.
Volume — Some questions benefit from huge amounts of data, with the sheer volume of data, it negates small messiness or inaccuracies.
Velocity — Real-time information → make swift decisions based on updated and informed predictions
Variety — Ability to ask new questions and form new connections, questions that were previously inaccessible
Veracity — Messy and unstructured data give rise to the possibility of hidden correlations.
Perhaps the most promising benefit of more data is to identify hidden correlations.
Examples:
A popular language model that uses Deep Learning. It has 175 billion parameters, it was built by eating up data from the internet to discover patterns and correlation. It’s capable of writing snippets of code,
The concept of Big data can be applied to this pandemic situation as well, by collecting data on the whereabouts of people (interactions and visited locations) with contact tracing, analytics can be done to predict the spread of the virus, and help contain it.
Having lots of Data has its benefits, but it doesn’t come without any challenges.
Lots of raw data to store and analyze
expensive and require good computing investment
data is constantly changing and fluctuating, systems built to handle that has to be adaptive
difficult to determine which source of data useful
notables to quickly analyze
need to clean data first
Big Data is commonly associated with other buzzwords like Machine Learning, Data Science, AI, Deep Learning, etc. Since these fields require data, Big data will continue to play a huge role in improving the current models we have now and allow for advancements in research. Take Tesla, for example, each Tesla car that has self-driving is also at the same time training Tesla’s AI model and continually improves it with each mistake. This huge siphoning of data allows, along with a team of talented engineers is what makes Tesla the best at the self-driving game.
As data continues to expand and grow, cloud storage providers like AWS, Microsoft Azure and Google Cloud will rule in storing big data. This allows room for scalability and efficiency for companies. This also means there will be more and more people hired to handle these data, which translate to more job opportunities for “data officers” to manage the database of a company
The future of Big data also has it’s dark sides, as you know, many tech companies are facing heat from governments and the public due to issues of privacy and data. Laws that govern the rights of the people to their data will make data collection more restricted albeit honest. By the same vein, the proliferation of data online also exposes us to cyberattacks, and data security will be incredibly important.
Many big tech companies today are receiving tons of data from its users, and when it comes down to profit and power or the greater good of society, it’s human nature to go for the former instead, especially if you’re in a position to choose. We live in times where our attention is being capitalized constantly. We must live smarter and act rationally to prevent surrendering our lives over to these short bursts of dopamine and expedient and trivial acts.
We can only hope that as we progress into the upcoming decades, the people who are in control of the decisions that these companies make will be for the betterment of society and civilization as a whole. And that our data will be for building systems that serve us, make us more productive, and instead of looking for ways to grab our attention, build products that can provide value and meaning to our lives.
A quick summary of everything so far
Big data — a large dataset of diverse types that are generated rapidly. There is a transition in data from structured to unstructured.
Unstructured Data — Data that isn’t clearly defined and requires more work to process (images, audio, social media likes, etc)
Structured Data — Known as traditional data as it is rare in real life, basically data that is clearly defined and easy to process. It’s data scientists’ job is to clean it and form tidy data
Pros & Cons in terms of 4Vs
Volume
slightly messy data negate small errors
a lot to store and analyze
Variety
answer unconventional questions
the burden of choice of type
Velocity
real-time analysis and decision making
constantly updating
Veracity
hidden correlations
Analyze messy data
One important lesson you should take away is that even with huge sums of data, you still need the right ones with the right variables to correctly answer your question.
A quote by John Turkey, a famous American mathematician, puts that lesson nicely:
The combination of some data and an aching desire for an answer does not ensure the a reasonable answer can be extracted from a given body of data — John Turkey, 1986
And another quote by Atul Butte, Stanford on the hidden capabilities of data
“Hiding within those mounds of data is knowledge that could change the life of a patient, or change the world.”
Thanks for reading and that concludes the series. I hope you learned something from the articles and do leave comments about how I can improve, and if you have any suggestions on what I should write next. Stay safe and God Bless.
medium.com
towardsdatascience.com
towardsdatascience.com
medium.com
towardsdatascience.com
If you want to be updated with my latest articles follow me on Medium.
Follow my other social profiles too!
Linkedin
Twitter
GitHub
Reddit
Be on the lookout for my next article and remember to stay safe!