AWS Bigdata Project Part 2
Channel: BigDatapedia ML & DS
Date: 2020-01-07

AWS Bigdata Project Part 2 - This channel is specially created and dedicated for the bigdata hadoop and it's ecosystem like spark ( pyspark and scala spark ) , hive , sqoop , hbase , kafka , flume , nifi , airflow with complete hands on traning in tamil language and focusing data science and Machine Learning technology in tamil and english hands on training. We also conducting trainings for bigdata certification in tamil and english like cloudera CCA175 and hortonworks also we cover up with bigdata and data science projects with real time stream processing and batch processing analytics with complete tamil and english support who want to learn from basics to advanced concepts. English classes also will cover in separate batch. 

===================
    WhatsApp :   https://chat.whatsapp.com/J10e2Ju7LhGLNv6VE5Lede
===================

***Download Airlines Dataset ***
http://stat-computing.org/dataexpo/2009/the-data.html


*** Codes ***
from pyspark.sql import SparkSession
from pyspark.sql.functions import *

spark = SparkSession.builder.appName("AirlinesDataset").getOrCreate()
df = spark.read.orc("s3://awsairlines2019/input/part-r-00000-7b23c7eb-1460-412a-b712-4574f4662771.orc")

df.printSchema()

#df.count()

print("\n Total Columns " len(df.columns))

df1 = df.select("month","flightnum","distance")

df2 = df1.filter(df1.month.isNotNull()).groupby("month","flightnum").agg({"distance":"sum"}).orderBy("month","flightnum").withColumnRenamed("sum(distance)","Total_Distance")

df2.printSchema()

print("\n Writing JDBC scenario_1 " )
sqlproperties = {"user":"admin", "password":"mysqladmin", "driver":"com.mysql.jdbc.Driver"}
df2.write.jdbc(url="jdbc:mysql//database-1.ccxirrlc2fnq.us-east-1.rds.amazonaws.com/rdsairlines",table="scenario_1",properties=sqlproperties)

# Scenario_2
df3 = df.select("month","flightnum","diverted")
df4 = df3.filter((df3.month.isNotNull()) & (df3.diverted=='1')).groupby("month","flightnum").agg({"diverted":"sum"}).orderBy("month","flightnum").withColumnRenamed("sum(diverted)","Total_diverted")

print("\n Writing JDBC scenario_2 " )
df4.write.jdbc(url="jdbc:mysql//database-1.ccxirrlc2fnq.us-east-1.rds.amazonaws.com/rdsairlines",table="scenario_2",properties=sqlproperties)

=====================================================


***~ BigDatapedia Forum ~***
*Telegram Link:
https://t.me/bigdatapedia

*WhatsApp Link:
https://chat.whatsapp.com/Ir9NhRq7NyX7MEKFaacJvR


*1. Spark interview Questions with Answers* 

http://bigdatascholars.blogspot.com/2018/07/spark-interview-questions.html

*2. Bigdata Interview Questions with Answers*

http://bigdatascholars.blogspot.com/2018/07/bigdata-interview-questions.html

*3.Hive Interview Questions with Answers*

http://bigdatascholars.blogspot.com/2018/08/hive-interview-questions-with-answers.html

*4.SQOOP Interview Questions with Answers*

http://bigdatascholars.blogspot.com/2018/08/sqoop-interview-question-and-answers.html

For Bigdata, Cloudera Certification and Datascience Training in Tamil or English 
contact (online and class room)
Dineshkumar S (Cloudera Certified) 
Mobile: 9715 010 010 / 9789888424

free Bigdata demo session on Every Saturday n providing interview discussions as well

Thanks
Bigdatapedia
Mobile # 9715 010 010


#bigdataproject #sparkprojectinaws #hadoopprojectinaws #hadoopaws #awsbigdataanalytics #dataengineeringaws #sparkwithaws #apachesparkinaws #datascientists #dataanalyst #datascientist #streaming #linux #dataanalysis #dataanalytics #datascience #bigdata #analytics #kafka #machinelearning #big_data #bigdataworld #bigdataengineer #bigdata2020 #bigdatahadoop #bigdataexpo #dataengineer #dataanalysts #dataengineers  #streaminganalytics #cloudera #datamining #apachehive #pyspark