Essentials skills for data science
By: Sajjad Hussain
Tags: Technology, Machine Learning, Data Science, Big Data, Tech

Data science requires us to think more systematically, combining a creative approach to the problem with the actual situation. This is like a way of thinking like a good architect, combining the artistic perspective (through design) with the project itself and time management. Planning is essential for the use of big data. Although it is the same task, different execution methods will involve greatly different requirements for resources, and the final results may not be much different.

The new thinking brought about by data scientists is obvious. Data scientists usually work in different teams composed of data modelers, business people, and other professionals (depending on different industries). Correspondingly, they hardly work for a long time alone like programmers who experience the traditional waterfall model.

In addition, data scientists can gradually solve the problem by consulting existing literature and establishing contacts with people who are familiar with the different fields that need to be solved, and breaking down the problem into small problems that can be handled.

MapReduce is a parallel and distributed algorithm used to divide the responsible task into a series of simplified tasks, and then use a very efficient means to solve them, based on this can increase the performance of processing complex tasks and reduce the cost of computing resources . Although this algorithm has existed before, its widespread use in data science has increased its popularity.

is a parallel and distributed algorithm used to divide the responsible task into a series of simplified tasks, and then use a very efficient means to solve them, based on this can increase the performance of processing complex tasks and reduce the cost of computing resources . Although this algorithm has existed before, its widespread use in data science has increased its popularity. Hadoop Distributed File System (HDFS) is an open source platform designed to make full use of parallel computing technology. It basically divides big data into small pieces and distributes them to computers on the network to achieve management.

(HDFS) is an open source platform designed to make full use of parallel computing technology. It basically divides big data into small pieces and distributes them to computers on the network to achieve management. Advanced text analysis often refers to natural language processing (NLP), which is a field of data information, including many techniques for extracting effective information and business intelligence from unstructured text data. Before data science, this field did not exist.

often refers to natural language processing (NLP), which is a field of data information, including many techniques for extracting effective information and business intelligence from unstructured text data. Before data science, this field did not exist. Large-scale data programming languages (such as Pig, R, ECL, etc.) are programming languages ​​that can effectively process large data sets. They were under development before the advent of data science or did not exist at all.

(such as Pig, R, ECL, etc.) are programming languages ​​that can effectively process large data sets. They were under development before the advent of data science or did not exist at all. Alternative database structures (for example, Hbase, Cassandra, MongoDB, etc.) are databases that use parallel computing technology to process big data archiving, querying, and editing.

You may be familiar with the New Technology File System (NTFS) used in modern Windows operating systems. This is a file system that is not problematic on most personal computers and is quite satisfactory. But it is not suitable for processing large amounts of data in an interconnected computer network. In addition, NTFS has a file size limit of 256 TB, which is far from enough for many big data applications. UNIX-based file systems face similar challenges, and this is why when Hadoop is developed, a new file system must be created, a file system suitable for computer clusters. HDFS allows users to browse all files on a computer cluster and can perform some basic operations as if they were operating on a separate computer (although most files may be spread across the entire network).

The soul of Hadoop is MapReduce, which is a rule that enables computer networks to efficiently split data with low risk. All data will be replicated to prevent the computers (usually called nodes) in the cluster from crashing. There are also some management nodes responsible for task scheduling and data flow management. First, all data will be mapped through a group of cluster nodes called Mappers. Once the execution is over, another group of nodes will accept the Reduce task. When the previously processed data is processed into useful output, this group of nodes is called a Reducer, or the task will eventually pass through the Mapper. . All things are coordinated through the management node. The management node can ensure that the output of all stages is safely stored on the cluster (in the form of multiple copies). Once the entire processing flow is over, the user will get the final output result. MapReduce rules contain many rather monotonous procedures. Its huge advantage is that it can ensure that the process will be completed faster, and efficiently use all available resources, while reducing the risk of data loss due to hardware failure (this is very common for large clusters).

Text analysis has been around for a while, but data science has introduced some advanced techniques that dwarf the previous techniques. Modern (advanced) text analysis allows users to process a large amount of text data, allowing for problems such as spelling errors, multi-word segmentation within sentences, and other problems, to accurately and quickly find text patterns. Advanced text analysis can determine emotional expressions on social media posts, and identify whether someone’s comments are straightforward or mocking. These are extremely difficult for machines to process without using these advanced text analysis methods. This advancement makes it possible to deploy artificial intelligence algorithm applications in the Hadoop environment.

Large-scale data programming languages, such as Pig, R, and ECL, are developed to solve big data problems, and they can be well integrated with the Hadoop environment (in fact, Pig is a part of the Hadoop ecosystem). R existed before the advent of big data and has undergone a series of major changes. It can now be connected with Hadoop and can process files in HDFS at the same time. Since programming languages ​​are not very difficult to develop in modern times, it is very likely that new programming languages ​​will appear in the ranks when you read this book, so there is always no harm in keeping your eyes open. By the end of this decade, it is likely that these existing languages ​​will not become the language of choice for data scientists (although the other language may still be widely circulated, based on its huge user base now).

Thanks to data science, new alternative database structures will arrive. These structures include hash tables (for example, JBoss Data Grid, Riak), B-trees (for example, MongoDB, CouchDB), and merge trees of log structures (for example, HBase, Cassandra). Unlike traditional databases, these pattern types are designed to serve big data, so the way they read and write data in the database is more flexible. Each type has its applicable and inapplicable categories. Since traditional databases will crash when the records exceed a certain level, they are easier to use than traditional databases. For example, if you have a large database with millions of rows and columns (such as a large data warehouse), finding the maximum value in a certain column in such a database may exceed the time many people are willing to wait. The same query in a columnar database (such as HBase) takes less than 1 second.

All these rules are based on the concept of a group of computers, that is, a computer cluster, which performs far better than any independent (or even super) computer, as long as the cluster is large enough. This kind of creativity is based on smart and customized strategies to plan basic tasks so that they can be efficiently processed by computer clusters, mainly to optimize the process of processing problems. Since its inception, the popularity of these new rules has continued to increase, and they have evolved rapidly. There is a lot of attention (and money) invested in these technologies. Learning from them will surely be rewarded in the near future.