I am working on the file "Python 복습/11일차.목_Pandas/11일차_1T_Pandas 의 자료형 - Series, DataFrame.ipynb".
The first task is:
위와 같은 방법은 좋은 게 아니다. 가급적 한글은 빼서 쓰자.
헤더에서는 영어를 쓰는 것이 좋다. 한글은 깨지기도 하고 문제가 생길 수 있다.
Python2에서 깨질 수 있다.

-
-시리즈의 기능으로 각각 채워 넣어진다.
Can you write Python code for it?

# DataFrame 에서 Row 를 가져오는 방법
df.loc[0]   #loc = location의 약자

df.Name[0]
#이것과 loc가 다른 것은? 리스트인데 딕셔너리처럼 동작도 가능한 리스트다.

df.loc[2]["Email"]

df.Name   #index로 나오게 되고

df.loc[2]   #column 이름으로 나오게 된다.


I am working on the file "Python 복습/11일차.목_Pandas/11일차_3T_ajax 로 이루어진 select, option 정보 가져오기 ( feat, 건강보험심사평가원 ).ipynb".
The first task is:
주소를 동적으로 ( ajax / javascript ) 를 이용해서 보내주는구나.
다음에는? 네트워크를 확인한다. 동적이 아닐 수도 있으니까. 정적일수도 있으니까
Can you write Python code for it?

response = requests.post("http://www.hira.or.kr/rd/hosp/selectSidoListAjax.do")

response.text


I am working on the file "Python 복습/12일차.금_Pandas의 고급기능_DB/12일차_1T_Pandas Basic (1) - 데이터 추가하기 ( pd.concat, df.append ).ipynb".
The first task is:
A와 B 회사가 합병해서 하나의 DF으로 만들자.
Can you write Python code for it?

a_df + b_df    # Matrix Sum 과 유사하게 합쳐진다. (행렬합처럼)


I am working on the file "Python 복습/12일차.금_Pandas의 고급기능_DB/12일차_1T_Pandas Basic (1) - 데이터 추가하기 ( pd.concat, df.append ).ipynb".
The first task is:
pd.concat으로 인덱스 기준으로 상위 하위 개념 나눌 수 있다.
Can you write Python code for it?

ab_df = pd.concat([a_df, b_df], keys=["KaKao", "Daum"])

ab_df.loc["KaKao"]   # 결과가 DataFrame 으로 나왔다.

a_df.loc[0]   #결과가 Series로 나온다. 원본 데이터 형태에 따라서 결과가 다르게 나온다.

ab_df.loc["KaKao"].loc[0]


I am working on the file "Python 복습/12일차.금_Pandas의 고급기능_DB/12일차_1T_Pandas Basic (1) - 데이터 추가하기 ( pd.concat, df.append ).ipynb".
The first task is:
위에서 Row로 합치는 것을 했다.
이제는 Column을 합치는 과정에 대해서. concat을 쓸 거야
Can you write Python code for it?

name_email_df = pd.DataFrame([
        {
            "Name": "김기표",
            "Email": "kkp@gmail.com",
        },
        {
            "Name": "유영수",
            "Email": "yys@naver.com",
        }
    ])

address_df = pd.DataFrame([
        {
            "Address": "패캠 대기빌딩 김기표님 앞",
        },
        {
            "Address": "패캠 대기빌딩 유영수님 앞",
        }
    ])

name_email_df

address_df

name_email_df + address_df

pd.concat([name_email_df, address_df])


I am working on the file "Python 복습/12일차.금_Pandas의 고급기능_DB/12일차_3T_Pandas Basic (3) - 데이터 그룹화 ( df.groupby ).ipynb".
The first task is:
각각의 df으로 만들고 싶다. (서울df, 부산df, 경북df)
for문을 돌리는 방법
Can you write Python code for it?

city_name = "서울"
is_city = df["시"] == city_name  # 특정 True/False의 Series를 만들고
df[is_city]                      # 해당하는 row만 뽑는다.

city_df_dict = {}

for city_name in df["시"].unique():
    is_city = df["시"] == city_name
    city_df = df[is_city]

    city_df_dict[city_name] = city_df

city_df_dict["부산"]   #Python의 자료형을 이용한 방법


I am working on the file "Python 복습/12일차.금_Pandas의 고급기능_DB/12일차_3T_Pandas Basic (3) - 데이터 그룹화 ( df.groupby ).ipynb".
The first task is:
중복되지 않는 “시”의 리스트
Can you write Python code for it?

df["시"]   #이거는 Series

list(df["시"])

set(list(df["시"]))

list(set(list(df["시"])))

#위의 과정을 압축하면
df["시"].unique()


I am working on the file "Python 복습/12일차.금_Pandas의 고급기능_DB/12일차_4T_Pandas Basic (4) - 파일 입출력 ( csv, excel, sql ).ipynb".
The first task is:
데이터 분석을 사용할 때 2가지 양식이 있다
csv(엑셀), XML, JSON == 데이터베이스
Pickle(데이터 분석에서 상당히 중요하다) => 파이썬의 객체 그대로 저장할 수 있다.       파이썬 코드를 그대로 저장
즉, 클래스나 함수를 바이너리 형태로 저장해서 언제든 쓸 수 있도록
Can you write Python code for it?

df = pd.read_csv
# 이렇게 간단하다
# 엑셀 파일을 일괄적으로 csv 형태로 바꿔주는 프로그래밍 => Pandas로 하면 금방 한다
# read_excel().to_csv 이런 식으로 하면. 해보자

pd.read_csv("fastcampus.csv")

pd.read_csv("fastcampus.csv", header=None, sep="|")

df = pd.read_csv("fastcampus.csv", header=None, sep="|")
df.rename(columns={0: "Age", 1: "Name"}, inplace=True)
df


I am working on the file "Python 복습/12일차.금_Pandas의 고급기능_DB/12일차_5T_Pandas 실습하기 - World 데이터베이스를 이용한 데이터 분석 ( filtering, merge, text mining ).ipynb".
The first task is:
1. 인구가 1억 명 이상이면서, Asia에 포함된 국가 리스트 DataFrame
Can you write Python code for it?

is_population = country_df["Population"] > 100000000
is_asia = country_df["Continent"] == "Asia"
df = country_df[is_population][is_asia]

# 번외로 정렬과 수치연산을 해보겠다.
df

# 정렬은 sort_values()
df.sort_values("Population", ascending=False)

# 수치연산은 꽤 쉽다.
df["Population"].min()
#min, max, sum, mean ...

df["Population"].mean()

#아시아의 국가별 평균 인구수
country_df[is_asia]["Population"].mean()


I am working on the file "Python 복습/12일차.금_Pandas의 고급기능_DB/12일차_5T_Pandas 실습하기 - World 데이터베이스를 이용한 데이터 분석 ( filtering, merge, text mining ).ipynb".
The first task is:
2. 국가명과 도시명을 가지고 있는 DataFrame(현재는 각각 country_df, city_df에 있는 상태)
Can you write Python code for it?

city_df.merge(country_df, left_on="CountryCode", right_on="Code")

city_df.merge(country_df, left_on="CountryCode", right_on="Code").rename(columns={
        "Name_x": "City",
        "Name_y": "Country",
    })[["City", "Country"]]


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_1T_Pandas로 배우는 SQL 시작하기 (1) - WHERE, ORDER BY.ipynb".
The first task is:
2. pandas (2)
Can you write Python code for it?

is_asia_or_europe = country_df["Continent"].isin(["Asia", "Europe"])
country_df[is_asia_or_europe].head(3)

# df["Continent"].str.contains("Asia")  =>  이건 텍스트마이닝 방법


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_1T_Pandas로 배우는 SQL 시작하기 (1) - WHERE, ORDER BY.ipynb".
The first task is:
3. sql (1)
Can you write Python code for it?

SQL_QUERY = """
    SELECT *
    FROM Country
    WHERE
        Continent = "Asia"
        OR Continent = "Europe"
"""

pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_1T_Pandas로 배우는 SQL 시작하기 (1) - WHERE, ORDER BY.ipynb".
The first task is:
4. sql (2)
Can you write Python code for it?

SQL_QUERY = """
    SELECT *
    FROM Country
    WHERE
        Continent IN ("Asia", "Europe")
    ;
"""

pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_1T_Pandas로 배우는 SQL 시작하기 (1) - WHERE, ORDER BY.ipynb".
The first task is:
Asia나 혹은 Europe에 속하는 국가를 1.pandas 2.sql 둘 다 뽑아보기
정답 4가지
1. pandas (1)
Can you write Python code for it?

is_asia = country_df["Continent"] == "Asia"
is_europe = country_df["Continent"] == "Europe"

country_df[is_asia | is_europe]


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_1T_Pandas로 배우는 SQL 시작하기 (1) - WHERE, ORDER BY.ipynb".
The first task is:
Name => 이름
Poplation => 인구
Continent = > 대륙으로 변경
Can you write Python code for it?

npc_df = country_df[["Name", "Population", "Continent"]]

npc_df.rename(columns={"Name": "이름", "Population": "인구", "Continent": "대륙"})

SQL_QUERY = """
    SELECT Name AS "이름", Population AS "인구수", Continent AS "대륙"
    FROM Country
    ;
"""

pd.read_sql(SQL_QUERY, db)

SQL_QUERY = """
    SELECT Name "이름", Population "인구수", Continent "대륙"
    FROM Country
    ;
"""
# AS가 없어도 동일하게 동작한다. 이건 없이 써도 됩니다.
pd.read_sql(SQL_QUERY, db)

country_df.count()

len(country_df)

SQL_QUERY = """
    SELECT COUNT(*)
    FROM Country;
"""

pd.read_sql(SQL_QUERY, db)

SQL_QUERY = """
    SELECT COUNT(*) "count"
    FROM Country;
"""

pd.read_sql(SQL_QUERY, db)

SQL_QUERY = """
    SELECT SUM(Population)
    FROM Country;
"""
#SQL에는 SUM, AVG 등 다양한 내장함수가 있다.
pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_1T_Pandas로 배우는 SQL 시작하기 (1) - WHERE, ORDER BY.ipynb".
The first task is:
SQL은 이런 연산에 굉장히 최적화 되어 있기 때문에 다른 언어보다 빠르다.
대신에 느린 부분은?
데이터가 100만개라면. Name, Population 등을 뽑을 때 다운로드 받아야 한다.
네트워크에 대한 비용이 걸린다. 다운 받는 속도 때문에 오래 걸린다.
그래서 빠르게 하는 방법은? SQL에서 가능한 연산은 SQL에서 돌리고 받는 것이 좋다.

조건에 따른 Filtering

Continent == Europe
Population > 10,000,000 데이터를 뽑아라
Can you write Python code for it?

is_europe = country_df["Continent"] == "Europe"
is_population = country_df["Population"] > 10000000

country_df[is_europe][is_population]

# AND => 유럽에 있고, 인구가 1000만 이상인 애들
# OR => 유럽에 있거나, 인구가 1000만 이상인 애들
# country_df[is_europe & is_population]
# country_df[is_europe | is_population]

SQL_QUERY = """
    SELECT *
    FROM Country
    WHERE
        Continent = "Europe"
        AND Population > 10000000
    ;
"""

pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_1T_Pandas로 배우는 SQL 시작하기 (1) - WHERE, ORDER BY.ipynb".
The first task is:
만약 어떤 데이터의 숫자만을 세야 하는 상황이라면? Pandas or SQL? SQL이 좋다.
Can you write Python code for it?

import time

start_time = time.time()

#여기서부터 시작
count = len(pd.read_sql("SELECT * FROM Country;", db))
print(count)

end_time = time.time()

excute_time = end_time - start_time
print(excute_time)

start_time = time.time()

#여기서부터 시작
pd.read_sql("SELECT COUNT(*) FROM Country;", db)

end_time = time.time()

excute_time = end_time - start_time
print(excute_time)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_1T_Pandas로 배우는 SQL 시작하기 (1) - WHERE, ORDER BY.ipynb".
The first task is:
인구수가 1억 명 이상이면서, 아시아나 유럽에 포함된 국가들을 인구수 내림차순으로 출력하라 와 같은 문제도 풀 수 있다.
Can you write Python code for it?

SQL_QUERY = """
    SELECT Name, Population
    FROM Country
    ORDER BY Population DESC
    ;
"""
# 기본값으로 ASC
pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_1T_Pandas로 배우는 SQL 시작하기 (1) - WHERE, ORDER BY.ipynb".
The first task is:
정렬
Pandas에서 인구수 내림차순으로 정렬(즉, 인구가 많은 순서대로)
Can you write Python code for it?


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_2T_Pandas로 배우는 SQL 시작하기 (2) - JOIN ON.ipynb".
The first task is:
(1) Pandas_merge
Can you write Python code for it?

country_df.columns

city_df.columns

city_df.merge(country_df, right_on="Code", left_on="CountryCode")[["Name_x", "Name_y"]]


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_2T_Pandas로 배우는 SQL 시작하기 (2) - JOIN ON.ipynb".
The first task is:
2. JOIN을 이용한 방법(DB를 생각해주는 착한 방법)
Can you write Python code for it?

SQL_QUERY = """
    SELECT co.Name "Country Name", ci.Name "City Name"
    FROM Country co
        JOIN City ci
        ON co.code = ci.CountryCode
    ;
"""

pd.read_sql(SQL_QUERY, db).head(2)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_2T_Pandas로 배우는 SQL 시작하기 (2) - JOIN ON.ipynb".
The first task is:
599 * 603 = 361197
데이터를 받아오는 데 오랜 시간이 걸렸다.
Can you write Python code for it?

SQL_QUERY = """
    SELECT customer.first_name, customer.last_name, address.address
    FROM customer, address
    WHERE
        customer.address_id = address.address_id
    ;
"""

df = pd.read_sql(SQL_QUERY, db).head()


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_2T_Pandas로 배우는 SQL 시작하기 (2) - JOIN ON.ipynb".
The first task is:
JOIN(pandas에서는 merge 기능)

"sakila"로 들어와서 실습
Can you write Python code for it?

db = pymysql.connect(
    "db.fastcamp.us",
    "root",
    "dkstncks",
    "sakila",
    charset = "utf8",
)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_2T_Pandas로 배우는 SQL 시작하기 (2) - JOIN ON.ipynb".
The first task is:
address(주소), customer(이름)
pandas merge를 이용해서 유저의 이름과 주소가 같이 있는 DataFrame을 만들어보기
Can you write Python code for it?

customer_df = pd.read_sql("SELECT * FROM customer;", db)

address_df = pd.read_sql("SELECT * FROM address;", db)

customer_df.columns

address_df.columns

customer_df.merge(address_df, on="address_id")[["first_name", "last_name", "address"]]
# left_on="address_id"
# right_on="address_id"
# => on...


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_2T_Pandas로 배우는 SQL 시작하기 (2) - JOIN ON.ipynb".
The first task is:
customer 599개 address는 603개가 있는 상황
Can you write Python code for it?

SQL_QUERY = """
    SELECT COUNT(*)
    FROM customer, address
    ;
"""

pd.read_sql(SQL_QUERY, db).head()


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_2T_Pandas로 배우는 SQL 시작하기 (2) - JOIN ON.ipynb".
The first task is:
pandas보다 sql에서 JOIN을 활용
sql 1번과 2번 중 무엇을 써야 하는가? JOIN!! JOIN을 쓰면 코드가 한 눈에 안 들어온다.
SQL (1) - SELECT ____ WHERE
SQL (2) - SELECT _ FROM _ JOIN ____
굳이 JOIN을 안 쓰더라도 상관 없어. WHERE문만 써도 괜찮아

World DB에서 Country와 City를 합쳐라. (Country Name, City Name)
3가지 방법으로. SQL(where, join)과 Pandas(merge)
Can you write Python code for it?

db = pymysql.connect(
    "db.fastcamp.us",
    "root",
    "dkstncks",
    "world",
    charset='utf8',
)
country_df = pd.read_sql("SELECT * FROM Country;", db)
city_df = pd.read_sql("SELECT * FROM City;", db)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_2T_Pandas로 배우는 SQL 시작하기 (2) - JOIN ON.ipynb".
The first task is:
즉, 데이터 테이블을 합친 다음에 ( 모든 row에 대한 곱 )
WHERE 조건문을 통해서 우리가 원하는 데이터만 가져오자.
Can you write Python code for it?

SQL_QUERY = """
    SELECT customer.first_name, customer.last_name, address.address
    FROM customer
    JOIN address ON customer.address_id = address.address_id
    ;
"""

pd.read_sql(SQL_QUERY, db).head()

import time
start_time = time.time()

customer_df = pd.read_sql("SELECT * FROM customer;", db)
address_df = pd.read_sql("SELECT * FROM address;", db)
df = customer_df.merge(address_df, on="address_id")

end_time = time.time()
exec_time = end_time - start_time
print(exec_time)

start_time = time.time()

SQL_QUERY = """
    SELECT customer.first_name, customer.last_name, address.address
    FROM customer, address
    WHERE
        customer.address_id = address.address_id
    ;
"""
df = pd.read_sql(SQL_QUERY, db)

end_time = time.time()
exec_time = end_time - start_time
print(exec_time)

start_time = time.time()

SQL_QUERY = """
    SELECT customer.first_name, customer.last_name, address.address
    FROM customer
    JOIN address ON customer.address_id = address.address_id
    ;
"""
pd.read_sql(SQL_QUERY, db)

end_time = time.time()
exec_time = end_time - start_time
print(exec_time)


I am working on the file "Python 복습/13일차.목_pandas + SQL/13일차_3T_Pandas로 배우는 SQL 시작하기 (3) - GROUP BY.ipynb".
The first task is:
payment

월별 전체 매출, 평균 매출, 결제 갯수(전체 매출 = 평균 매출 * 결제 갯수)
pandas
Can you write Python code for it?

SQL_QUERY = """
    SELECT *
    FROM payment
    ;
"""
payment_df = pd.read_sql(SQL_QUERY, db)

payment_df = payment_df[["payment_date", "payment_id", "amount"]]

payment_df.head(1)

payment_df["month"] = payment_df["payment_date"].apply(lambda x: str(x)[:7])

payment_df.groupby("month").agg({"payment_id": np.size, "amount": [np.sum, np.mean]})

# 마지막으로 sql

SQL_QUERY = """
    SELECT
        LEFT(payment_date, 7) Month,
        count(*) "Total Payment",
        SUM(amount) "Total Revenue",
        AVG(amount) "Average Payment"
    FROM payment
    GROUP BY Month
    ;
"""

pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_1T_os, shutil 모듈을 이용한 파일,폴더 관리하기 (1) - 폴더 생성 및 제거.ipynb".
The first task is:
df.to_csv(os.path.join(, , ___.csv))
df.to_csv("./data/world/Angola.csv")
Can you write Python code for it?

# 폴더의 유무를 확인하고, 있으면 삭제한다.
if "data" in os.listdir():
    print("./data/ 폴더를 삭제합니다.")
    shutil.rmtree(os.path.join(os.curdir, "data"))

# "data"라는 폴더를 생성하고, 그 안에 "world"라는 폴더를 생성한다.
print("./data/ 폴더를 생성합니다.")
os.makedirs(os.path.join(os.curdir, "data"))
os.makedirs(os.path.join(os.curdir, "data", "world"))

import pymysql

db = pymysql.connect(
    "db.fastcamp.us",
    "root",
    "dkstncks",
    "world",
    charset='utf8'
)

country_df = pd.read_sql("SELECT * FROM Country;", db)
city_df = pd.read_sql("SELECT * FROM City;", db)
#Country.Code를 바탕으로, City.CountryCode와 매칭해서 찾아야 함
#Country.Name은 반드시 가지고 와야지 파일명으로 저장이 가능

city_groups = city_df.groupby("CountryCode")

for index, row in country_df.iterrows():
    country_code = row["Code"]
    country_name = row["Name"]

    city_df = city_groups.get_group(country_code)
    city_df.to_csv(os.path.join("data", "world", "{country_name},csv".format(country_name=country_name)))

#"ATA"라는 애가 없다고 나오니까 테스트
SQL_QUERY = """
    SELECT *
    FROM City
    WHERE CountryCode = "ATA"
    ;
"""
pd.read_sql(SQL_QUERY, db)

city_groups.get_group("ATA")

"ATA" in city_groups["CountryCode"].unique()

#없는게 증명 됐으니 if문 첨가
for index, row in country_df.iterrows():
    country_code = row["Code"]
    country_name = row["Name"]

    if country_code in city_df["CountryCode"].unique():
        one_city_df = city_groups.get_group(country_code)
        one_city_df.to_csv(os.path.join(os.curdir, "data", "world", "{country_name}.csv".format(country_name=country_name)))


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_1T_os, shutil 모듈을 이용한 파일,폴더 관리하기 (1) - 폴더 생성 및 제거.ipynb".
The first task is:
ipynb 라는 확장자로 끝나는 파일들만 가지고 오려면
Can you write Python code for it?

[
    file_name
    for file_name
    in os.listdir("../01일차.수_입문/")
    if file_name.endswith(".ipynb")   # csv 파일 가져오기, 엑셀 파일 가져오기로 사용
]


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_1T_os, shutil 모듈을 이용한 파일,폴더 관리하기 (1) - 폴더 생성 및 제거.ipynb".
The first task is:
os - low-level (저수준)으로 파일/폴더/운영체제를 관리했다면
shutil - high-level (고수준) 으로 파일/폴더를 관리
Can you write Python code for it?

os.listdir(os.path.join(os.curdir, "data"))

shutil.rmtree(os.path.join(os.curdir, "data"))

os.listdir(os.path.join(os.curdir, "data"))

os.makedirs(os.path.join(os.curdir, "data"))

shutil.rmtree(os.path.join(os.curdir, "data"))


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_1T_os, shutil 모듈을 이용한 파일,폴더 관리하기 (1) - 폴더 생성 및 제거.ipynb".
The first task is:
os.curdir     #current directory
os.path.join(...)
os.listdir(...)
Can you write Python code for it?

os.makedirs("data")   #잠재적인 문제가 있다.

os.listdir()   #폴더 만들기는 쉽게 됩니다.

os.rmdir("data")   #잠재적인 문제가 있다.

os.listdir()


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_1T_os, shutil 모듈을 이용한 파일,폴더 관리하기 (1) - 폴더 생성 및 제거.ipynb".
The first task is:
설정 파일 같은 것을 수정하거나 삭제할 때
만약에 .bash_profile => .bash_profile.tmp / ... (복사해주고 작업을 한다.)

복구는 안 된다.

위와 같은 과정의 flow는 어려워

그래서 shutil이라는 파이썬 내장 모듈을 사용할 것임
Can you write Python code for it?


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_1T_os, shutil 모듈을 이용한 파일,폴더 관리하기 (1) - 폴더 생성 및 제거.ipynb".
The first task is:
파일에 대한 경로를 생성할 때

현재 폴더 안에 있는, "data"라는 폴더의 "data.csv"의 경로
"data/data.csv"
"./data/data.csv" ( String 으로 입력할 때 이렇게 직접 ) // 무조건 이 방법으로
"/home/python/notebooks/dobestan/dss/....../data/data.csv"   - 절대 경로 // 잘 안 씀
Can you write Python code for it?

os.path.join("data", "data.csv")

os.curdir

os.path.join(os.curdir, "data", "data.csv")
# 이렇게 하면 경로를 알려줘. 앞으로 만들 때는 무조건 이렇게 만들겠다.
# os.path.join(os.curdir, "data", file_name)


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_2T_os, shutil 모듈을 이용한 파일,폴더 관리하기 (2) - 압축 파일 생성 및 해제.ipynb".
The first task is:
실습)

다시 실행해도 문제가 없이,
대륙명 / 국가명.csv로 City에 대한 정보가 저장.
대륙명.tar.gz (대륙명 별 압축파일로 저장)
/data/world/Asia/Korea.csv
/Japan.csv
...
/data/world/Europe/France.csv
Can you write Python code for it?

for index, row in country_df.iterrows():
    country_code = row["Code"]
    country_name = row["Name"]

    if country_code in city_df["CountryCode"].unique():
        one_city_df = city_groups.get_group(country_code)
        one_city_df.to_csv(os.path.join(os.curdir, "data", "world", "{country_name}.csv".format(country_name=country_name)))

import pymysql
db = pymysql.connect(
    "db.fastcamp.us",
    "root",
    "dkstncks",
    "world",
    charset='utf8',
)
city_df = pd.read_sql("SELECT * FROM City;", db)
country_df = pd.read_sql("SELECT * FROM Country;", db)

if "data" in os.listdir():
    print("./data/폴더를 삭제합니다.")
    shutil.rmtree(os.path.join(os.curdir, "data"))

print("./data/폴더를 생성합니다.")
os.makedirs(os.path.join(os.curdir, "data"))
os.makedirs(os.path.join(os.curdir, "data", "world"))

# country_df => group_by => continent
# cotinent 폴더 ...
# continent_df => group_by => ...

continent_groups = country_df.groupby("Continent")
city_groups = city_df.groupby("CountryCode")

# "ATA" ... => 예외처리(city_df가 없는 경우)
unique_country_code_in_city = city_df["CountryCode"].unique()

for continent_name in country_df["Continent"].unique():
    os.makedirs(os.path.join(os.curdir, "data", "world", continent_name))
    continent_df = continent_groups.get_group(continent_name)

    #continent_df (대륙별 DataFrame)에서 "Code"를 가져와서 city_groups에서 찾은 다음에 폴더에 넣어주자
    for index, row in continent_df.iterrows():
        country_code = row["Code"]
        country_name = row["Name"]

        if country_code in unique_country_code_in_city:
#             print((continent_name, country_name))
            df = city_groups.get_group(country_code)
            df.to_csv(os.path.join(
                os.curdir,
                "data",
                "world",
                continent_name,
                "{country_name}.csv".format(country_name=country_name)
                ))
    # 압축하기
    shutil.make_archive(
    os.path.join(os.curdir, "data", "world", continent_name),  # 압축파일명
    "gztar",
    os.path.join(os.curdir,...


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_2T_os, shutil 모듈을 이용한 파일,폴더 관리하기 (2) - 압축 파일 생성 및 해제.ipynb".
The first task is:
압축 푸는 방법
Can you write Python code for it?


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_3T_데이터베이스, 테이블 생성하고 데이터 추가하기.ipynb".
The first task is:
실습)

데이터를 어떻게 저장할 것인가? - 확장성이 있는가, ... ( JOIN, Merge, GROUP BY, ... )

provider ( "직방", "다방", "꿀방", "두더지방", ) - id, name

agency ( "부동산명" ) - id, provider_id, phonenumber, name
room ( "매물" ) - address, deposit, rent
Can you write Python code for it?

# 1. 데이터베이스 연결
SQL_QUERY = """
    USE kipoy;
"""
cursor.execute(SQL_QUERY)

# 2. 기존의 데이터베이스 제거
SQL_QUERY = """
    DROP TABLE IF EXISTS provider;
"""
cursor.execute(SQL_QUERY)

# 3. 테이블 생성하기
SQL_QUERY = """
    CREATE TABLE IF NOT EXISTS provider
    (
        provider_id int PRIMARY KEY,
        name varchar(20)
    )
    ;
"""
cursor.execute(SQL_QUERY)
db.commit()

SQL_QUERY = """
    SELECT *
    FROM provider
    ;
"""

pd.read_sql(SQL_QUERY, db)

SQL_QUERY = """
    INSERT INTO provider (provider_id, name)
    VALUES (
        1,
        "다방"
    );
"""
cursor.execute(SQL_QUERY)
db.commit()

pd.read_sql("SELECT * FROM provider;", db)   #받아오는 데 시간이 꽤 걸리네


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_4T_Pandas로 배우는 SQL 시작하기 (4) - HAVING, SUB QUERY.ipynb".
The first task is:
HAVING
Can you write Python code for it?

SQL_QUERY = """
    SELECT r.customer_id, COUNT(*) rentals
    FROM rental r
        JOIN customer c
        ON r.customer_id = c.customer_id
    GROUP BY r.customer_id
    # WHERE rentals > 30  #연산에 대한 결과로 Filtering을 할 수 없다.
    # 연산에 대한 결과로 Filtering을 할 수 있는 기능
    HAVING rentals > 30
"""

pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/14일차.금_pandas + SQL_2/14일차_4T_Pandas로 배우는 SQL 시작하기 (4) - HAVING, SUB QUERY.ipynb".
The first task is:
film_df => film_id, title
inventory_df => inventory_id, film_id
rental_df => rental_id, inventory_id
payment_df => rental_id, amount
Can you write Python code for it?

SQL_QUERY = """
    SELECT f.film_id, f.title, SUM(p.amount) "revenue"
    FROM
        film f,
        rental r,
        payment p,
        inventory i
    WHERE
        f.film_id = i.film_id
        AND i.inventory_id = r.inventory_id
        AND r.rental_id = p.rental_id
    GROUP BY f.film_id
    ORDER BY revenue DESC
    ;
"""
pd.read_sql(SQL_QUERY, db)

SQL_QUERY = """
    SELECT f.film_id, f.title, SUM(p.amount) "revenue"
    FROM payment p
        JOIN rental r ON p.rental_id = r.rental_id
        JOIN inventory i ON i.inventory_id = r.inventory_id
        JOIN film f ON f.film_id = i.film_id
    GROUP BY f.film_id
    ORDER BY revenue DESC
    ;
"""

pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/15일차.목_serialize, SQL실습/15일차_1T1_파이썬에서 객체를 serialize,deserialize 하는 방법 - json, pickle.ipynb".
The first task is:
json <<<<<<< pickle을 많이 쓴다.
Can you write Python code for it?

import pickle

pickle.dumps(users)

users_pickle = pickle.dumps(users)

type(users_pickle)

users_pickle   #python만 읽을 수 있는 형태

pickle.loads(users_pickle)   #다시 돌아온다.

pickle.dump(users, open("users.pkl", "wb"))

pickle.load(open("users.pkl", "rb"))

# 서브 노트(1T2)에서도 띄워 봤을 때 뜨는 지 확인했음

pickle.dumps(df)

pickle.dump(df, open("zigbang.pkl", "wb"))

# 다시 1T2로

# serialize => dump/dumps
# deserialize => load/loads

# json, pickle

# .csv / .excel / .pickle.dump 3가지 형태로 각각 다 저장해서 쓰면 좋다.

df.to_pickle("./topickle.pkl")

# 다시 1T2로


I am working on the file "Python 복습/15일차.목_serialize, SQL실습/15일차_2T_데이터 분석을 위한 SQL 실습 (1) - WHERE IN, LIKE, JOIN.ipynb".
The first task is:
film테이블에서 등급으로 그룹을 묶어서, 각 등급별 갯수, 평균 렌탈 비용을 모두 출력하세요
Can you write Python code for it?

SQL_QUERY = """
    SELECT rating, COUNT(*) "total_films", AVG(rental_rate) "average_rental_rate"
    FROM film
    GROUP BY rating
    ORDER BY average_rental_rate
    ;
"""
pd.read_sql(SQL_QUERY, db)

SQL_QUERY = """
    SELECT
        rating,
        COUNT(*) "total_films",
        AVG(rental_rate) "average_rental_rate"
    FROM film
    GROUP BY
        1
    ORDER BY 3
    ;
"""
pd.read_sql(SQL_QUERY, db)

film_df.groupby("rating").agg({
        "film_id": {"total films": np.size},
        "rental_rate": {"average_rental_rate": np.mean},
    })


I am working on the file "Python 복습/15일차.목_serialize, SQL실습/15일차_3T_데이터 분석을 위한 SQL 실습 (2) - SUB QUERY, HAVING.ipynb".
The first task is:
JOIN은 조금 어렵지만 속도가 WHERE보다 빠르다.
Can you write Python code for it?


I am working on the file "Python 복습/15일차.목_serialize, SQL실습/15일차_3T_데이터 분석을 위한 SQL 실습 (2) - SUB QUERY, HAVING.ipynb".
The first task is:
pandas
SUBQUERY - 1. Rental per User, 2.30개 이상인 애들 뽑기
Can you write Python code for it?

rc_df = rental_df.merge(customer_df, on="customer_id")

rc_df.groupby("customer_id").size() >= 30

rentals_per_customer_df = rc_df.groupby("customer_id").agg({"customer_id": np.size})

is_30 = rentals_per_customer_df.customer_id > 30
rentals_per_customer_df[is_30]


I am working on the file "Python 복습/15일차.목_serialize, SQL실습/15일차_5T_데이터 분석을 위한 SQL 실습 (4) - SQL Advanced.ipynb".
The first task is:
Store 1의 등급별 매출 중 "R", "PG-13"의 매출

film, payment, inventory, rental
Can you write Python code for it?

payment_df = pd.read_sql("SELECT * FROM payment;", db)

film_df.head(1)

payment_df.head(1)

inventory_df.head(1)

rental_df.head(1)

SQL_QUERY = """
    SELECT
        i.store_id store_id,
        f.rating rating,
        SUM(p.amount) total_revenue
    FROM
        payment p
            JOIN rental r ON p.rental_id = r.rental_id
            JOIN inventory i ON r.inventory_id = i.inventory_id
            JOIN film f ON i.film_id = f.film_id
    WHERE
        i.store_id = 1
        AND f.rating IN ("PG-13", "R")
    GROUP BY
        store_id,
        rating
    ;
"""
pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/15일차.목_serialize, SQL실습/15일차_5T_데이터 분석을 위한 SQL 실습 (4) - SQL Advanced.ipynb".
The first task is:
배우별 매출

영화별 매출을 구하고 이 데이터를 바탕으로 배우별 매출을 구하세요
Can you write Python code for it?

# 1. 영화별 매출 - rental, film, inventory
REVENUE_PER_FILM_SQL_QUERY = """
    SELECT
        f.film_id film_id,
        COUNT(*) * f.rental_rate revenue
    FROM
        rental r
            JOIN inventory i ON r.inventory_id = i.inventory_id
            JOIN film f ON i.film_id = f.film_id
    GROUP BY
        film_id
    ;
"""
pd.read_sql(REVENUE_PER_FILM_SQL_QUERY, db)

# 2. 배우별 매출 - actor, film_actor

SQL_QUERY = """
    SELECT
        a.actor_id,
        a.last_name last_name,
        a.first_name first_name,
        SUM(rpf.revenue) revenue_per_actor
    FROM ({REVENUE_PER_FILM_SQL_QUERY}) AS rpf
        JOIN film_actor fa ON rpf.film_id = fa.film_id
        JOIN actor a ON fa.actor_id = a.actor_id
    GROUP BY
        actor_id
    ORDER BY revenue_per_actor DESC
    ;
""".format(
    REVENUE_PER_FILM_SQL_QUERY=REVENUE_PER_FILM_SQL_QUERY.replace(";",""),
)
pd.read_sql(SQL_QUERY, db)


I am working on the file "Python 복습/16일차.금_nosql/16일차_2T_데이터 분석을 위한 MongoDB (2)_실습.ipynb".
The first task is:
Document 의 기준을 무엇으로 보는가? 우리는 아까 몽고DB에서 Country로 보았다

인구가 10만 이상인 도시 리스트를 뽑아올 수 있는가? 몽고 DB 쿼리 만으로는 안 된다.

왜냐하면 다큐먼트 하나하나의 기준이 country이기 때문에. 만약에 city를 기준으로 하려고 했다면 몽고 DB에서 차라리
이게 더 좋은 방법이다.
Can you write Python code for it?

city = {
    "name": "Seoul",
    "population" = 10000000,
    "country": {
        "name": "Korea",
        "population": 50000000
    }
}


I am working on the file "Python 복습/16일차.금_nosql/16일차_2T_데이터 분석을 위한 MongoDB (2)_실습.ipynb".
The first task is:
find_찾을 때
insert_넣을 때
Can you write Python code for it?

user = {
    "name": "이동규",
    "email": "milk@gmail.com",
}
users.insert_one(user)

users_list = [
    {
        "name": "이영은",
        "phone": "01022706512",
    },
    {
        "name": "유영수",
        "email": "yskk@naver.com"
    }
]
users.insert_many(users_list)

watcha = client2.kipoy["watcha"]

watcha = db.watcha

import requests

response = requests.get("https://watcha.net/home/news.json?page=1&per=300")

watcha_news = response.json().get("news") # List

watcha.insert_many(watcha_news)


I am working on the file "Python 복습/16일차.금_nosql/16일차_2T_데이터 분석을 위한 MongoDB (2)_실습.ipynb".
The first task is:
관계형 DB를 몽고DB로 옮길 때 어떻게 옮기는 것이 가장 좋을까?
pandas로 country city df 옮겨보자
Can you write Python code for it?

import pandas as pd
import pymysql

db = pymysql.connect(
    "db.fastcamp.us",
    "root",
    "dkstncks",
    "world",
    charset='utf8',
)

# db = MySQLdb.connect(
#     "192.168.0.199",
#     "root",
#     "rlarlvy",
#     "sakila",
#     charset='utf8',
# )
# => 이거 왜 안되는지 모르겠다.

country_df = pd.read_sql("SELECT * FROM Country;", db)
city_df = pd.read_sql("SELECT * FROM City;", db)

country_df.columns

city_df.columns

# List of countries.
[
    {
        "Name": "Republic of Korea",
        "cities": [ #List of cities
            {
                "Name": "Seoul",
                "Population": "100000000",
            },
            {
                "Name": "Busan",
            }

        ]
    }
]

country_df.to_json()

country_df.to_dict()

city_groups = city_df.groupby("CountryCode")

country_list = []

for index, row in country_df.iterrows():
    country_dict = {}

    country_code = row["Code"]
#     city_group_df = city_groups.get_group(country_code) 위랑 같아
    country_dict["name"] = row["Name"]
    country_dict["population"] = row["Population"]
    country_dict["cities"] = []  # cities 라는 key 로 빈 리스트를 초기화 한다.

    if country_code in city_df["CountryCode"].unique():
        city_group_df = city_groups.get_group(country_code)

        for city_index, city_row in city_group_df.iterrows():
            city_dict = {}
            city_dict["name"] = city_row["Name"]
            city_dict["population"] = city_row["Population"]
            country_dict["cities"].append(city_dict)

    country_list.append(country_dict)

country_list

len(country_list)

world = client.kipoy["world"]

world

world.insert_many(country_list)


I am working on the file "Python 복습/16일차.금_nosql/16일차_2T_데이터 분석을 위한 MongoDB (2)_실습.ipynb".
The first task is:
보증금이 1000만원 이상인 매물 가져오기
Can you write Python code for it?

MONGO_QUERY = {}
cursor = zigbang.find(MONGO_QUERY)

items_list = [
    document
    for document
    in cursor
    if document.get("item").get("deposit") >= 1000
]
len(items_list)

MONGO_QUERY = {"item.deposit": {"$gte": 1000}}   #1000보다 크거나 같다. greater than equal
zigbang.find(MONGO_QUERY).count()

MONGO_QUERY = {"item.deposit": {"$gte": 1000, "$lte": 2000}}
zigbang.find(MONGO_QUERY).count()


I am working on the file "Python 복습/16일차.금_nosql/16일차_2T_데이터 분석을 위한 MongoDB (2)_실습.ipynb".
The first task is:
보증금이 2000만원 이하이고 월세가 50만원 이하인 매물 출력하기
Can you write Python code for it?

MONGO_QUERY = {}
cursor = zigbang.find(MONGO_QUERY)

items_list = [
    document
    for document
    in cursor
    if document.get("item").get("deposit") <= 2000 and document.get("item").get("rent") <= 50
]
len(items_list)

MONGO_QUERY = {
    "item.deposit": {"$lte": 2000},
    "item.rent": {"$lte": 50},
    }
zigbang.find(MONGO_QUERY).count()

zigbang_list = zigbang_dict.get("items")

len(zigbang_list)

new_zigbang_list = [
    item.get("item")
    for item
    in zigbang_list
]

new_zigbang_list[0]

zigbang.insert_many(new_zigbang_list)

MONGO_QUERY = {"deposit": {"$lte": 1000}}
zigbang.find(MONGO_QUERY).count()


I am working on the file "Python 복습/16일차.금_nosql/16일차_2T_데이터 분석을 위한 MongoDB (2)_실습.ipynb".
The first task is:
보증금이 딱 1000만원인 매물 리스트를 출력해보자
Can you write Python code for it?

MONGO_QUERY = {}
cursor = zigbang.find(MONGO_QUERY)
items_list = [
    document
    for document
    in cursor
    if document.get("item").get("deposit") == 1000
]

len(items_list)

MONGO_QUERY = {"title": "서울시 관악구 신림동"}
zigbang.find_one(MONGO_QUERY)

MONGO_QUERY = {"item.deposit": 1000}   #관계형 DB의 관점에서는 말도 안되는 기능. 쉽게 찾을 수 있다.
zigbang.find(MONGO_QUERY).count()


I am working on the file "Python 복습/16일차.금_nosql/16일차_2T_데이터 분석을 위한 MongoDB (2)_실습.ipynb".
The first task is:
비정형데이터에서 중요한 것은 어떤 데이터를 잘 넣을까 하는 부분에 대해서 고민해보아야 한다.
왜냐하면 있는 그대로 넣었을 경우에 비효율적인 부분이 발생하기 때문이다.

3T_Aggregate ( GROUP BY )

GROUP BY ... 지역별 숫자 세기 / 지역별 평균 보증금,월세 ( MongoDB )

World => MongoDB ( 데이터 전처리 ... )

aggregate ( GROUP BY )

zigbang ... address1을 기준으로 나누어보자(그룹을 나누어보자)
Can you write Python code for it?

import pymongo

#GROUP BY
cursor = zigbang.aggregate([
        {
            "$group": {
                "_id": "$address1",
            }
        }
    ])
for document in cursor:
    print(document)

cursor = zigbang.aggregate([
        {
            "$group": {
                "_id": "$address1",
                "count": {"$sum": 1}
            }
        }
    ])
for document in cursor:
    print(document)

cursor = zigbang.aggregate([
        {
            "$group": {
                "_id": "$address1",
                "count": {"$sum": 1},
                "total deposit": {"$sum": "$deposit"}
            }
        }
    ])
for document in cursor:
    print(document)

cursor = zigbang.aggregate([
        {
            "$group": {
                "_id": "$address1",
                "count": {"$sum": 1},
                "average rent": {"$avg": "$rent"},
            }
        }
    ])
for document in cursor:
    print(document)

cursor = zigbang.aggregate([
        {
            "$group": {
                "_id": "$address1",
                "count": {"$sum": 1},
                "average rent": {"$avg": "$rent"},
                "average deposit": {"$avg": "$deposit"}
            }
        }
    ])
for document in cursor:
    print(document)

db = client.dobestan

db.collection_names()

restaurants = db.restaurants   #MONGO DB에서 제공하는 데이터

restaurants.find().count()

cursor = restaurants.aggregate([
    {
       "$group": {
            "_id": "$borough",
        }
    }
])
for document in cursor:
    print(document)

cursor = restaurants.aggregate([
    {
       "$group": {
            "_id": "$borough",
                "count": {"$sum": 1},
        }
    }
])
for document in cursor:
    print(document)


I am working on the file "Python%2BTutorial-Template.ipynb".
The first task is:
Conditionals in Python
Can you write Python code for it?

#Sometimes you want to execute code only in certain circumstances. We saw this on HW1.
#The basic syntax is:
#if boolean_statement_1:
#    _code to do if 1 is true_
#elif boolean_statement_2:
#    _code to do if 2 is true_
#else:
#    _code to do in all other cases_

#A boolean statement is something that can be True or False. Named after mathematician George boole. (5 < 3) is an
#example, and in this case evaluates to False.

#The tabs are mandatory. Let's do some more examples

answer = 43
if answer == 42:
    print("This is the answer")
    print("No it really is")
elif answer < 42:
    print("This is too low")
else:
    print("This is too high")
print("This is always run")

rainy = True
day = "Tuesday"

if(rainy == False) and (day != "Tuesday"):
    print("Price is $10")
elif(rainy == True) and (day == "Tuesday"):
    print("Price is $5")
elif(rainy == True) or (day == "Tuesday"):
    print("Price is $7.50")


I am working on the file "Python%2BTutorial-Template.ipynb".
The first task is:
Functions in Python
Can you write Python code for it?

#We can separate off code into functions, that can take input and can give output. They serve as black boxes from the
#perspective of the rest of our code

#use the def keyword, and indent because this creates a new block
def print_me( string ):
    print(string)

    #End with the "return" keyword
    return

print_me("My name is Jacob")

def my_favorite_song():
    ans = "Amsterdam - Imgaine Dragons"
    return ans

print(my_favorite_song())
#What if we want to compute something and return the result? How would you write that return statement?


I am working on the file "Python%2BTutorial-Template.ipynb".
The first task is:
Numpy - "The Fundamental Package for Scientific Computing with Python"
Can you write Python code for it?

import numpy as np
#Here, we grab all of the functions and tools from the numpy package and store them in a local variable called np.
#You can call that variable whatever you like, but 'np' is standard.

#numpy has arrays, which function similarly to python lists.
a = np.array( [1,2,3] )
b = np.array( [9,8,7] )
#Be careful with syntax. The parentheses and brackets are both required!
print(a)

#Access elements from them just like you would a regular list
print(a[1:3])

#Element-wise operations are a breeze!
c = a + b
d = a - b
e = a * b
f = a / b

g = np.matmul(a,b)
print(g)

#What are other things you would like to do with these arrays?

#If we don't get to it, be sure to look at the notes once uploaded for Numeric ODE Integration!
def deriv(y,t):
    return t

IC = 0

start_time = 0
end_time = 10
num_times = 101

times = np.linspace(start_time,end_time, num_times)
print(times)

from scipy.integrate import odeint

integrated_vector = odeint(deriv,IC,times)

%matplotlib inline
import matplotlib.pyplot as plt
from ipywidgets import interact

plt.plot(times,integrated_vector)
plt.title("y = (1/2)t^2")


I am working on the file "Python+Operators.ipynb".
The first task is:
'+ is the operator symbol, 2 and 3 are the operands
Operator Types

Arithmetic Operators
Comparison Operators
Logical Operators
Bitwise Operators
Assignment Operators
Special Operators

Arithmetic Operators
Can you write Python code for it?

x = 15
y = 6

# Addition Operator
print('x + y = ', x + y)

# Subtraction Operator
print('x - y = ', x - y)

# Multiplication Operator
print('x * y = ', x * y)

# Division Operator
print('x / y = ', x / y) # True Division
print('x // y =', x//y) # Class Division

# Exponential Operator
print('x ** y = ', x ** y)


I am working on the file "Python+Operators.ipynb".
The first task is:
Assignment Operators
Can you write Python code for it?


I am working on the file "Python+Operators.ipynb".
The first task is:
Membership Operators
<pre>
    in - True if value / variable is found in sequence
    not in - True if value / variable is not found in the sequence
</pre>
Can you write Python code for it?

x = 'Hello World'
y = {1:'a',2:'b'}

print("'H' in x ", 'H' in x)
print('hello not in x ','hello' not in x)
print('1 in y = ',1 in y)
print('a in y = ',a in y)


I am working on the file "Python+Operators.ipynb".
The first task is:
Special Operators
Identity Operators
<pre>
 is - True if the operands are identical
 is not - True if the operands are not identical
 </pre>
Can you write Python code for it?


I am working on the file "Python+Postgres.ipynb".
The first task is:
%sql also makes it easy to create new tables (but CREATE TABLE ... AS SELECT ... would be far more efficmient)
Can you write Python code for it?

%sql PERSIST df

%sql SELECT * FROM df

r


I am working on the file "Python+Postgres.ipynb".
The first task is:
%sql can't see these updates yet though...
Can you write Python code for it?

%sql SELECT * FROM df

c2.commit()

c2.connection.commit()
c2.connection is db

%sql SELECT * FROM df


I am working on the file "Python+Postgres.ipynb".
The first task is:
%sql does use it's own, separate connection though, so it can't see anything that's not yet committed.
Can you write Python code for it?

%sql SELECT * FROM pg_tables WHERE tableowner='decibel'

db.commit()

%sql SELECT * FROM pg_tables WHERE tableowner='decibel'

c = db.cursor()

c.execute("SELECT * FROM pg_tables LIMIT 5")

r = c.fetchone()

r

for result in c: # Note that c is our cursor
    print result

SQLactivity="SELECT pid,datname,usename,application_name,state_change,state,query FROM pg_stat_activity"
c.execute(SQLactivity)

c.fetchmany(2)

c.statusmessage

c.rownumber

c.rowcount

len(c.fetchall())


I am working on the file "Python+Postgres.ipynb".
The first task is:
Because the sql magic uses SQLalchemy under the hood, it can end up in a transaction, which means you might get stale data. If that happens, you want to issue either a ROLLBACK; or COMMIT;.
Can you write Python code for it?

%%sql
ROLLBACK;
SELECT pid,datname,usename,application_name,state_change,state,query FROM pg_stat_activity;


I am working on the file "Python+Postgres.ipynb".
The first task is:
Results from c2 would have names associated with them, but...
Can you write Python code for it?

c2.execute(SQLactivity)
pd.DataFrame([row for row in c2])

c2.execute(SQLactivity)
pd.DataFrame([dict(row) for row in c2])

new_df=pd.read_sql(SQLactivity, db)
new_df

%sql SELECT pid,datname,usename,application_name,state_change,state,query FROM pg_stat_activity

c2_df_result

%sql SELECT * FROM df

# Lets create a new table...
c2.execute('CREATE TABLE df2(s serial, index int, schemaname name, count bigint)')

# And insert a row into it
SQLdf2_insert='INSERT INTO df2 VALUES(DEFAULT, %(index)s, %(schemaname)s, %(count)s)'
c2.execute(SQLdf2_insert, c2_df_result[0])

%sql SELECT * FROM df2

db.commit()

%sql SELECT * FROM df2

c.executemany(SQLdf2_insert, c2_df_result) # Note executemany(), not just execute()
pd.read_sql('SELECT * FROM df2', db)

c.execute('SELECT * FROM df2')
c.fetchall()

c3=db.cursor(cursor_factory=psycopg2.extras.NamedTupleCursor)
c3.execute('SELECT * FROM df2')
r=c3.fetchall()
r

r[0]

r[0][0]

r[0].s

r[0][1] is r[0].index

c3.execute(SQLdf2_insert + ' RETURNING *', r[0]._asdict())
c3.fetchall()

%sql SELECT * FROM df2

with db:
    c3.execute(SQLdf2_insert + ' RETURNING *', r[0]._asdict())
    r=c3.fetchall()

r

%sql SELECT * FROM df2

c.execute('BAD SQL')

c.execute('SELECT 1').fetchall()

db.rollback()

with db:
    c.executemany(SQLdf2_insert, c2_df_result)
    c.execute('BAD SQL')
    c.execute('SELECT 1')
    c.fetchall()

c.execute('SELECT 1')
c.fetchall()

with db.cursor() as new_c:
    new_c.execute('SELECT * FROM df2')
    r=new_c.fetchall()

r

new_c

new_c.execute('SELECT 1')


I am working on the file "Python+Postgres.ipynb".
The first task is:
We can easily turn tuple results into a dataframe, but the dataframe won't have any names.
Can you write Python code for it?


I am working on the file "Python+Postgres.ipynb".
The first task is:
We can haz DataFrames
Can you write Python code for it?

import pandas as pd

df = r.DataFrame()

df

r


I am working on the file "Python+Postgres.ipynb".
The first task is:
and plots/graphs
Can you write Python code for it?


I am working on the file "Python+Postgres.ipynb".
The first task is:
c2_df_result is unchanged though...
Can you write Python code for it?


I am working on the file "Python+Postgres.ipynb".
The first task is:
psycopg2 supports non-standard types
Can you write Python code for it?

c.execute("CREATE TYPE card AS (value text, suit text);")

c.execute("SELECT ('Ace', 'Spades')::card")
r=c.fetchone()[0]
r

type(r)

psycopg2.extras.register_composite('card', c)
c.execute("SELECT ('Ace', 'Spades')::card")
r=c.fetchone()[0]
r

type(r)

%sql CREATE TYPE complex AS(r float, i float)

c3.execute('SELECT row(.1,.2)::complex')
r=c3.fetchone()
r

type(r)

type(r.row)

z=complex(2,3)
z

c2.execute('SELECT %s',(z,))
r=c2.fetchone()
r

import psycopg2.extensions as exten

def adapt_complex(c):
    return exten.AsIs("(%s,%s)" % (c.real, c.imag))

exten.register_adapter(complex,adapt_complex)

c.execute('CREATE TABLE c AS SELECT %s::complex AS c', (z,))

c.execute('INSERT INTO c VALUES(%s),(%s)',(z * 2, z.conjugate()))
c2.execute('SELECT * FROM c')
r=c2.fetchall()
db.commit()
r

type(r[0][0])

%sql SELECT * FROM c

%sql SELECT (c).* FROM c

import re
def cast_complex(value, cursor):
    if value is None:
        return None

    # Match a value that looks like (foo,bar)
    m = re.match(r"\(([^)]+),([^)]+)\)", value)
    if m:
        return complex(float(m.group(1)),float(m.group(2)))
    else:
        raise InterfaceError("bad representation for type complex: %r" % value)

c2.description[0]

c2.description[0].type_code

complex_caster = exten.new_type((c2.description[0].type_code,), 'complex', cast_complex)

complex_caster

exten.register_type(complex_caster)

c2.execute('SELECT * FROM c')
r=c2.fetchall()
r

type(r[0][0])

%sql SELECT * FROM c


I am working on the file "Python-01.ipynb".
The first task is:
Flow Controls
if else, for
Can you write Python code for it?

if 2>3:
    # This is inside of loop
    print('this')
    print('this')
    print('this')
    if 2<5:
        print('2<5')
else:
    print('that')
    print('that')
print("end")

list(range(3,10,2))

for i in range(10):
    print(i)

for i in l:
    print(i)

for i in [0, 0.1, 0.2, 0.3]:
    print(i)

for i in range(10):
    print(i*0.1)


I am working on the file "Python-01.ipynb".
The first task is:
Input and outputs
print(), '{0:8.4}'.format(), input(), eval()
Can you write Python code for it?

print(l)

print(l,ln,ln1)

'b is {1} and a is {0}, {2:5.3}.'.format(a,b, 0.3333333333)

a = input("Engter somthing:")

a

eval(a)

eval('math.sqrt(2)')


I am working on the file "Python-01.ipynb".
The first task is:
List
in, not in, +, *, [], len(), min(), max(), sum(), help()
Can you write Python code for it?


I am working on the file "Python-01.ipynb".
The first task is:
Variable names
good practice: temp_var, tempVar, TempVar, unicode, = , a,b = b,a
Can you write Python code for it?

a = 3

a

a,b = 3,4

a

b

a,b = b,a

a

b

حسین = 3

حسین + b

3 = a


I am working on the file "Python-01.ipynb".
The first task is:
function
comments, docstrings, numbers and lists
Can you write Python code for it?

def f(x):
    '''
    Add one to x and return it.
    This is a simple function to demonstrate a concept.
    '''
    x+=1
    return x

help(f)

a = 3

f(a)

a

def g(l):
    l[0] = 'saeed'
    return l

l

g(l)

l

help(g)


I am working on the file "Python-02.ipynb".
The first task is:
Dictionaries
items get keys pop update values
Can you write Python code for it?

d = {}
d['name'] = 'Hamed'
d['family name'] = 'Seyed-allaei'
d[0]=12
d['a']=''
print(d)
print(d['name'])
print(d[0])

for i,j in d.items():
    print(i,j)


I am working on the file "Python-02.ipynb".
The first task is:
Generators
next()
Can you write Python code for it?

def myrange(n):
    i = 0
    while i < n:
        yield i
        yield i**2
        i+=1

x = myrange(10)
type(x)

next(x)

[i for i in myrange(10)]

for i in myrange(10):
    print(i)


I am working on the file "Python-02.ipynb".
The first task is:
Queez
Once upon a time, there was king, who wanted lots of soldiers. So he commanded every couple in the country to have children, until their first dauter born. Then the family is banned from having any more child.
What will be the ratio of boy/girls in this country?
Can you write Python code for it?

from random import randint

children = 0
boy = 0

for i in range(10000):
    gender = randint(0,1) # boy=1, girl=0
    children += 1
    while gender != 0:
        boy += gender
        gender = randint(0,1)
        children += 1
print(boy/children)


I am working on the file "Python-02.ipynb".
The first task is:
S be carefull!
while Loop
Can you write Python code for it?


I am working on the file "Python-4 Dictionary and Set.ipynb".
The first task is:
Dictionary and Set
Dictionary and Set are also container type data. They can contain any data type, including other continaer type objects.
Dictionary and Set are Mutable objects.
Dictionary
dictionary[key] = value will create or update the value in the dictionary.
Can you write Python code for it?

x = dict()  # Create empty dictionary
x[0] = 'abc'  # Create
x[0] = 'xyz'  # update
print(x[0])
print(x[1])  # No item... error

# Dicationary which value is List
# Disctionary is enclosed with brace {}

x = {'a': ['apple', 'anaconda'], 'b': ['beatles', 'beethoven']}
x['b'].append('beegees')
print(x)

# MCPC rehearsal problem 1

s1 = "Living John Piano"
s2 = "Bath Alice Toothbrush"

who_what_dict = dict()    # Key Where, Value (Who, What)
x = s1.split()
who_what_dict[x[0]] = x[1:]
x = s2.split()
who_what_dict[x[0]] = x[1:]

# What is in Living ?
print(who_what_dict['Living'][1])    # [0]: Who, [1]: What


I am working on the file "Python-4 Dictionary and Set.ipynb".
The first task is:
Operators of Set
Product (AND), Union (OR), Exclusive OR (XOR) and Difference are supported as oprators to created evaluated Set. Super Set, Sub Set oprators returns Boolean value.

AND:  &
OR:   |
XOR:  ^
Difference: -
Superset >
Subset <
Can you write Python code for it?

set_a = {1, 2, 3, 4}
set_b = {3, 4, 5, 6}

print(set_a & set_b)
print(set_a | set_b)
print(set_a ^ set_b)
print(set_a - set_b)

set_a = {1, 2, 3, 4}
set_b = {3, 4}
set_c = {3, 4}

print (set_a > set_b)   # is set_a a superset of set_b ?
print (set_b > set_a)
print (set_b < set_a)
print (set_b >= set_c)
print (set_b > set_c)


I am working on the file "Python-4 Dictionary and Set.ipynb".
The first task is:
Set
Set is like a dictionary without value (only key), or List of unique items.

Elements of Set can be accessed as a iterable object (can be used in for loop)
Elements of Set can not be accessed with Index, (same as Dictionary)
String, List, Tuple or other iterable object can be converted to Set
Can you write Python code for it?

list_x = [1, 1, 2, 2, 3, 3, 4]
set_x = set(list_x)
print(set_x)

str_x = 'this is a string'
set_x = set(str_x)
print(set_x)

print(set_x[0])   # error


I am working on the file "Python-5 Class and object.ipynb".
The first task is:
Class usage sample (Binary Tree)
Can you write Python code for it?

# 2017 MMCPC rehearsal problem-D

class BinTreeNode():
    def __init__(self, val):
        self.val = val
        self.parent = None
        self.left = None
        self.right = None

    def add_left(self, left):
        left.parent = self
        self.left = left

    def add_right(self, right):
        right.parent = self
        self.right = right

    def add_child(self, child):
        if child.val < self.val:
            if self.left == None:
                self.add_left(child)
            else:
                self.left.add_child(child)
        else:
            if self.right == None:
                self.add_right(child)
            else:
                self.right.add_child(child)

    def print_node(self):
        if self.parent: print('Parent:', self.parent.val, end=' ')
        if self.left: print('Left:', self.left.val, end = ' ')
        if self.right: print('Right', self.right.val, end = ' ')
        print('Node val:', self.val)
        if self.left: self.left.print_node()
        if self.right: self.right.print_node()

root_node = BinTreeNode(11)

for val in (6, 19, 4, 8, 17, 43, 5, 10, 31,49):
    node = BinTreeNode(val)
    root_node.add_child(node)

root_node.print_node()


I am working on the file "Python-5 Class and object.ipynb".
The first task is:
Special method of class

__init__(self, ...)
__lt__
__repr__
__add__
...
Can you write Python code for it?

#!/usr/bin/python3
# -*- coding: utf-8 -*-

class rectangle():
    def __init__(self, size_x, size_y):
        self.size_x = size_x
        self.size_y = size_y
        self.area = size_x * size_y

    def __lt__(self, other):
        return self.area < other.area

    def __repr__(self):
        return ('Rectangle('+str(self.size_x)+'x'+str(self.size_y)+')')

sq1 = rectangle(1,11)
sq2 = rectangle(3,4)
sq3 = rectangle(2,5)

x = [sq1, sq2, sq3]
x.sort()
print(x)


I am working on the file "Python-7 Input and Output.ipynb".
The first task is:
Data conversion
int() function convert string or float to integer, str() convert to string, float() convert to float.
map(func, iterable-object) apply function to iterable-object and returns iterable object.
Can you write Python code for it?

s = '100'
print(int(s)+1)

s = '1 2 3'
for i in map(int, s.split()):
    print(i)

s = '1 2 3 4'
x = list(map(int, s.split()))
print(x)

y = list()
for i in s.split():  # ['1', '2', '3', '4']
    y.append(int(i))

print(y)


I am working on the file "Python-7 Input and Output.ipynb".
The first task is:
I/O
File I/O
Similar to fopen() in C, open() close() read() write() seek() are supported, not only these function calls, readline() readlines() are also supported.
File object can be treated as a iterator, may be it is more common.
print() function can write to file with file= option. open(), close(), readline(), print() will be most commonly used function for text I/O.
readline() function returns the 1 line data include \n, and returns '' (0 length string) for EOF (End Of File).
Can you write Python code for it?

fd = open('README.md', 'r')
print(fd.readline(), end='')   # \n is included in input string

for s in fd:      # file object(descriptor) is iterable, and can be used in for loop
    print(s.strip())    # strip() removes extra space and \n
    # print(s.split())  # convert string to List


I am working on the file "Python-7 Input and Output.ipynb".
The first task is:
Standard In, Standard Out and Standard Error
sys.stdin, sys.stdout and sys.stderr are file descriptors which are already opened for application programs.
stardard-in is assinged to keyboard, standard-out and standard-error are assinged to display as a default. These assigned can be overrided when executing the program from shell.
$ program_name <input_file
$ program_name >output_file
$ program_name <input_file >output_file
$ program_name 2>error_file
Can you write Python code for it?

# need to import sys module to use standard I/O file descriptor
import sys
print('input somthing: ')

# jupyter does not handle stdin ?
# sys.stdin is used as a file-descriptor
s = sys.stdin.readline()
print(s.split())   # convert input to List


I am working on the file "Python-Crash-Course/Python Crash Course Exercises .ipynb".
The first task is:
Create a function that counts the number of times the word "dog" occurs in a string. Again ignore edge cases.
Can you write Python code for it?

def countDog(inp):
    dog = 0
    for x in inp.lower().split():
        if x == 'dog':
            dog += 1
    return dog

countDog('This dog runs faster than the other dog dude!')


I am working on the file "Python-Crash-Course/Python Crash Course Exercises .ipynb".
The first task is:
Create a function that grabs the email website domain from a string in the form:
user@domain.com

So for example, passing "user@domain.com" would return: domain.com
Can you write Python code for it?

def domainGet(inp):
    return inp.split('@')[1]

domainGet('user@domain.com')


I am working on the file "Python-Crash-Course/Python Crash Course Exercises .ipynb".
The first task is:
Final Problem
You are driving a little too fast, and a police officer stops you. Write a function
  to return one of 3 possible results: "No ticket", "Small ticket", or "Big Ticket".
  If your speed is 60 or less, the result is "No Ticket". If speed is between 61
  and 80 inclusive, the result is "Small Ticket". If speed is 81 or more, the result is "Big    Ticket". Unless it is your birthday (encoded as a boolean value in the parameters of the function) -- on your birthday, your speed can be 5 higher in all
  cases.
Can you write Python code for it?

def caught_speeding(speed, is_birthday):
    if is_birthday:
        speed = speed - 5
    if speed > 80:
        return 'Big Ticket'
    elif speed > 60:
        return 'Small Ticket'
    else:
        return 'No Ticket'

caught_speeding(81,True)

caught_speeding(81,False)


I am working on the file "Python-Crash-Course/Python Crash Course Exercises .ipynb".
The first task is:
Given this nested dictionary grab the word "hello". Be prepared, this will be annoying/tricky
Can you write Python code for it?


I am working on the file "Python-Crash-Course/Python Crash Course Exercises .ipynb".
The first task is:
Given this nested list, use indexing to grab the word "hello"
Can you write Python code for it?


I am working on the file "Python-Crash-Course/Python Crash Course Exercises .ipynb".
The first task is:
Use lambda expressions and the filter() function to filter out words from a list that don't start with the letter 's'. For example:
seq = ['soup','dog','salad','cat','great']

should be filtered down to:
['soup','salad']
Can you write Python code for it?

seq = ['soup','dog','salad','cat','great']

list(filter(lambda item:item[0]=='s',seq))


I am working on the file "Python-Crash-Course/Python Crash Course Exercises .ipynb".
The first task is:
What is the main difference between a tuple and a list?
Can you write Python code for it?

# Tuple is immutable, list items can be changed


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
---------------------------------------------
TASK: Clip band 4 of the satellite image so that it covers cloud-free areas in the Turku archipelago (in the North-East corner of the scene).
You can open the band 4 manually in QGIS for defining the corner coordinates.
---------------------------------------------
Next, let's repeat the clipping for all the rest of the bands all at once. For doing this, we will use Python for generating the command for each spectral band in our scene.
Can you write Python code for it?

import glob
import os

# List filepaths for all bands in the scence
FileList = glob.glob(os.path.join(r'/home/geo/LandsatData','*band*.tif'))

# Define clipping extent
xmin, ymin, xmax,ymax = (0, 0, 0, 0) # INSERT HERE THE CORRECT COORDINATES

# Generate gdalwarp command for each band
command = ""

for fp in FileList:
    inputfile = fp
    outputfile = inputfile[:-4] + "_clip.tif"

    command += "gdalwarp -te %s %s %s %s %s %s \n" % (xmin, ymin, xmax, ymax, inputfile, outputfile)

# Write the commands to an .sh file
cmd_file = "ClipTurkufromLandsat.sh"
f = open(os.path.join(cmd_file), 'w')

f.write(command)
f.close()


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
1. Open a raster file in Python with GDAL
With GDAL, you can read and write several different raster formats in Python. Python automatically registers all known GDAL drivers for reading supported formats when the importing the GDAL module. Most common file formats include for example TIFF and GeoTIFF, ASCII Grid and Erdas Imagine .img -files.
Landsat 8 bands are stored as separate GeoTIFF -files in the original package. Each band contains information of surface reflectance from different ranges of the electromagnetic spectrum. You can read more information about Landsat 8 here.
Let's start with inspecting one of the files we downloaded:
Can you write Python code for it?

from osgeo import gdal

filepath = r"LandsatData/LC81910182016153LGN00_sr_band4.tif"

# Open the file:
raster = gdal.Open(filepath)

# Check type of the variable 'raster'
type(raster)


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
As you can see, our we have now stored the same raster data into two different types of variables:
-raster is a Gdal Dataset
-rasterArray is a numpy array
The GDAL library also comes with a module gdal_array that works as an interface between NumPy and GDAL. Gdal_array reads and writes raster files to and from NumPy arrays directly from file:
Can you write Python code for it?

from osgeo import gdal_array
#from osgeo import gdalnumeric

# Read raster data as numeric array from file
rasterArray = gdal_array.LoadFile(filepath)


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
As you can see, the numpy array still contains the original nodata values. Any calculations will be wrong if these are not taken care of.
Can you write Python code for it?

import numpy as np

# Get nodata value from the GDAL band object
nodata = band.GetNoDataValue()

#Create a masked array for making calculations without nodata values
rasterArray = np.ma.masked_equal(rasterArray, nodata)
type(rasterArray)

# Check again array statistics
rasterArray.min()


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
Get raster bands
In our case, all bands of the Landsat 8 scene are stored as separate files. rasterCount is 1 as we have only opened one GeoTiff containing Landsat 8 band 4. However, different bands of a satellite images are often stacket together in one raster dataset in which case rasterCount would be greater than one.
In order to have a closer look at the values stoden in the band, we will take advantage of the GDAL Band API.
Can you write Python code for it?

# Read the raster band as separate variable
band = raster.GetRasterBand(1)

# Check type of the variable 'band'
type(band)

# Data type of the values
gdal.GetDataTypeName(band.DataType)


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
NOTE: If you are working in an windows environment, change the .sh extension to .bat.
After running the above script, you should have a file "ClipTurkufromLandsat.sh" in your working directory. Open the file (with a text editor) and check that the commands have been written correctly to the file.
Next, run the file in the Terminal window:
Can you write Python code for it?


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
Now we have a GDAL Raster Band object stored in the variable band.
Data type of the band can be interpreted with the help of GDAL documentation on Pixel data types. Unsigned integer is always equal or greater than zero and signed integer can store also negative values. For example, an unsigned 16-bit integer can store 2^16 (=65,536) values ranging from 0 to 65,535.
Band  statistics
Next, let's have a look at the values that are stored in the band. You might need to calculate statistics for the raster before being able to print out any information.
Can you write Python code for it?

# Compute statistics if needed
if band.GetMinimum() is None or band.GetMaximum()is None:
    band.ComputeStatistics(0)
    print("Statistics computed.")

# Fetch metadata for the band
band.GetMetadata()

# Print only selected metadata:
print ("[ NO DATA VALUE ] = ", band.GetNoDataValue()) # none
print ("[ MIN ] = ", band.GetMinimum())
print ("[ MAX ] = ", band.GetMaximum())


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
Now you have a two-dimensional array ready for further calculations. However, for completing our exercise for this week we will use a very simple command line tool gdal_calc.py.
Close raster dataset
It might be useful to close an exitsing GDAL object in the middle of the code if you want to free resources and remove unnecessary varables from memory.
Can you write Python code for it?


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
Now you should have a bunch of clipped .tif files ready and you might want to open a few of them in QGIS to check that the process was succesful.
Generating a layer stack with gdal_merge.py
After clipping the image you can for example stack bands 3 (green), 4 (red), and 5 (nir) for visualizing a false-color composite. Merge the layers with gdal_merge.py and use the -separate option for indicating that you wish to save the inputs as separate bands in the output file.
Let's try running the command as a subprocess in python:
Can you write Python code for it?

import os

# Define input and output files
inputfiles =  "band3_clip.tif band4_clip.tif band5_clip.tif"
outputfile =  "Landsat8_GreenRedNir.tif"

# Generate the command
command = "gdal_merge.py -separate %s -o %s" % (inputfiles, outputfile)

# Run the command. os.system() returns value zero if the command was executed succesfully
os.system(command)


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
Read raster file properties
The satellite image is now stored as a GDAL Dataset object in the variable raster. Let's have a closer look at the properties of the file:
Can you write Python code for it?

# Projection
raster.GetProjection()

# Dimensions
raster.RasterXSize
raster.RasterYSize

# Number of bands
raster.RasterCount

# Metadata for the raster dataset
raster.GetMetadata()


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
Take care of NoData values
Can you write Python code for it?

# What is the minimum value of the array?
rasterArray.min() #-9999


I am working on the file "Python-and-Gdal.ipynb".
The first task is:
so what's the difference?
Can you write Python code for it?

#Check the datatype of variables
type(rasterArray)
type(raster)


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise - Solutions.ipynb".
The first task is:
Create a 3x3 identity matrix
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise - Solutions.ipynb".
The first task is:
Create a 3x3 matrix with values ranging from 0 to 8
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise - Solutions.ipynb".
The first task is:
Create an array of 10 fives
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise - Solutions.ipynb".
The first task is:
Create an array of 20 linearly spaced points between 0 and 1:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise - Solutions.ipynb".
The first task is:
Create an array of the integers from 10 to 50
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise - Solutions.ipynb".
The first task is:
Create the following matrix:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise - Solutions.ipynb".
The first task is:
Get the sum of all the columns in mat
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise - Solutions.ipynb".
The first task is:
Now do the following
Get the sum of all the values in mat
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise - Solutions.ipynb".
The first task is:
Numpy Indexing and Selection
Now you will be given a few matrices, and be asked to replicate the resulting matrix outputs:
Can you write Python code for it?

mat = np.arange(1,26).reshape(5,5)
mat

# WRITE CODE HERE THAT REPRODUCES THE OUTPUT OF THE CELL BELOW
# BE CAREFUL NOT TO RUN THE CELL BELOW, OTHERWISE YOU WON'T
# BE ABLE TO SEE THE OUTPUT ANY MORE

mat[2:,1:]

# WRITE CODE HERE THAT REPRODUCES THE OUTPUT OF THE CELL BELOW
# BE CAREFUL NOT TO RUN THE CELL BELOW, OTHERWISE YOU WON'T
# BE ABLE TO SEE THE OUTPUT ANY MORE

mat[3,4]

# WRITE CODE HERE THAT REPRODUCES THE OUTPUT OF THE CELL BELOW
# BE CAREFUL NOT TO RUN THE CELL BELOW, OTHERWISE YOU WON'T
# BE ABLE TO SEE THE OUTPUT ANY MORE

mat[:3,1:2]

# WRITE CODE HERE THAT REPRODUCES THE OUTPUT OF THE CELL BELOW
# BE CAREFUL NOT TO RUN THE CELL BELOW, OTHERWISE YOU WON'T
# BE ABLE TO SEE THE OUTPUT ANY MORE

mat[4,:]

# WRITE CODE HERE THAT REPRODUCES THE OUTPUT OF THE CELL BELOW
# BE CAREFUL NOT TO RUN THE CELL BELOW, OTHERWISE YOU WON'T
# BE ABLE TO SEE THE OUTPUT ANY MORE

mat[3:5,:]


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Create a 3x3 identity matrix
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Create a 3x3 matrix with values ranging from 0 to 8
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Create an array of 10 fives
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Create an array of 10 ones
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Create an array of 10 zeros
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Create an array of 20 linearly spaced points between 0 and 1:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Create an array of the integers from 10 to 50
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Create the following matrix:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Get the sum of all the columns in mat
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Now do the following
Get the sum of all the values in mat
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Use NumPy to generate a random number between 0 and 1
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/NumPy/Numpy Exercise .ipynb".
The first task is:
Use NumPy to generate an array of 25 random numbers sampled from a standard normal distribution
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/DataFrames.ipynb".
The first task is:
Can also drop rows this way:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/DataFrames.ipynb".
The first task is:
Conditional Selection
An important feature of pandas is conditional selection using bracket notation, very similar to numpy:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/DataFrames.ipynb".
The first task is:
DataFrame Columns are just Series
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/DataFrames.ipynb".
The first task is:
For two conditions you can use | and & with parenthesis:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/DataFrames.ipynb".
The first task is:
Multi-Index and Index Hierarchy
Let us go over how to work with Multi-Index, first we'll create a quick example of what a Multi-Indexed DataFrame would look like:
Can you write Python code for it?

# Index Levels
outside = ['G1','G1','G1','G2','G2','G2']
inside = [1,2,3,1,2,3]
hier_index = list(zip(outside,inside))
hier_index = pd.MultiIndex.from_tuples(hier_index)

hier_index

df = pd.DataFrame(np.random.randn(6,2),index=hier_index,columns=['A','B'])
df


I am working on the file "Python-for-Data-Analysis/Pandas/DataFrames.ipynb".
The first task is:
Now let's show how to index this! For index hierarchy we use df.loc[], if this was on the columns axis, you would just use normal bracket notation df[]. Calling one level of the index returns the sub-dataframe:
Can you write Python code for it?

df.loc['G1']

df.loc['G1'].loc[1]

df.index.names

df.index.names = ['Group','Num']

df

df.xs('G1')

df.xs(['G1',1])

df.xs(1,level='Num')


I am working on the file "Python-for-Data-Analysis/Pandas/DataFrames.ipynb".
The first task is:
Selecting Rows
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/DataFrames.ipynb".
The first task is:
Selecting subset of rows and columns
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/Groupby.ipynb".
The first task is:
And then call aggregate methods off the object:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/Groupby.ipynb".
The first task is:
Now you can use the .groupby() method to group rows together based off of a column name. For instance let's group based off of Company. This will create a DataFrameGroupBy object:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/Merging, Joining, and Concatenating .ipynb".
The first task is:
Concatenation
Concatenation basically glues together DataFrames. Keep in mind that dimensions should match along the axis you are concatenating on. You can use pd.concat and pass in a list of DataFrames to concatenate together:
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/Merging, Joining, and Concatenating .ipynb".
The first task is:
Example DataFrames
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/Merging, Joining, and Concatenating .ipynb".
The first task is:
Joining
Joining is a convenient method for combining the columns of two potentially differently-indexed DataFrames into a single result DataFrame.
Can you write Python code for it?


I am working on the file "Python-for-Data-Analysis/Pandas/Merging, Joining, and Concatenating .ipynb".
The first task is:
Merging
The merge function allows you to merge DataFrames together using a similar logic as merging SQL Tables together. For example:
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Below is another example of printing the first ten observations from the CASTable object tbl
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Calculating simple statistics for the CASTable object.
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Create a new CAS Table Using SQL
Use the CREATE TABLE syntax in SQL.
Can you write Python code for it?

conn.fedsql.execDirect(query='''
    create table cars_temp_SQL as
    select *
    from cars_temp
    where EngineSize>5
    ''')

conn.tableinfo()


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Exercise 3 - Running CAS Actions
Now that a CAS connection has been established, we can try running some CAS actions.
CAS actions are called on CAS objects just like Python methods. In the example below, we will use the serverstatus action to display some basic status information about the CAS server such as platform and release information, the number of nodes, and information about each node in the grid. That information will be stored in an object called out.
The result of every CAS action is a CASResults object. CASResults objects are simply a subclass of Python's ordered dictionary, so anything that you can do with Python dictionaries can also be done with CASResults objects.
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Exercise 6 - Building Analytical Models
Regression Model
The first step is to load a modeling action set before using it. Let's load action sets for regression and decision tree models.
Can you write Python code for it?

conn.loadactionset('regression')

conn.help(actionset='regression')

glm_out=tbl.glm(
    target = 'MSRP',
    nominals = ['Type', 'Origin'],
    inputs = ['Type', 'Origin', 'MPG_City', 'Length', 'Weight']
)

glm_out


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
How do you delete a CAS table?
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Now that the 'simple' Action set has been loaded, we can use the 'correlation' action within the 'simple' Action set to calculate a Pearson correlation matrix.
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Now the CAS server has two tables loaded onto it. This can be verified using the tableinfo() function and the CASConnection object.
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
ODS-like Output Style
Can you write Python code for it?

from swat.render import render_html
render_html(glm_out)


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Plotting
There are several packages for creating plots in Python. One of those packages is matplotlib. Using matplotlib, we can create a scatter plot of MSRP values from the DataFrame we created.
Can you write Python code for it?

%matplotlib inline

df.plot(kind='scatter', x='MSRP', y='Horsepower', figsize=(12,6))


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Random forest models are commonly used in variable selection. Selection is usually determined by the
variable importance of the predictors in training the forest model. In the foresttrain action, this
importance measure is defined as the total Gini reduction from all of the splits that use this predictor. You
can request variable importance using the varimp option and generate the variable importance plot using the
Matplotlib package.
Can you write Python code for it?

forest1.varimp = True
result = forest1()
dfVarImp = result['DTreeVarImpInfo']
import matplotlib.pyplot as plt
import numpy as np
y_pos = np.arange(len(dfVarImp['Importance']))
plt.barh(y_pos, dfVarImp['Importance'], align='center')
plt.yticks(y_pos, dfVarImp['Variable'])
plt.xlabel('Variable Importance')
plt.show()


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Running SQL Code
SQL stands for Structured Query Language, and is commonly used when working with relational database management systems. In this section, we will illustrate how to use SQL to communicate to the CAS server.
The first step is to load the FedSQL actionset.
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Simple Statistics
One way to calculate simple statistics is to calculate multiple simple statistics for specific columns.
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Subsetting DataFrames
You can do various types of data manipulation on DataFrames. One example is to create a new DataFrame containing only selected columns of an existing DataFrame.
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Suppose I want a new DataFrame containing only models with an MSRP greater than 80,000.
Can you write Python code for it?

exp=df['MSRP'] > 80000

exp.head(15)

df80=df[exp]

df80.head()


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
The function that reads CSV files into DataFrames is called read_csv. In the simplest form, you supply it with a filename or URL. The cars data set is stored as a CSV in github. We'll reference the URL, and use the head() method to print the first several observations.
Can you write Python code for it?

df=pd.read_csv('https://raw.githubusercontent.com/'
               'sassoftware/sas-viya-programming/master/data/cars.csv')

df.head(10)


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
The models in the decisiontree action set support either continuous, binary or multilevel response
variable. Let us fit a random forest model to predict whether a vehicle is from Asia, Europe, or United
States.
The forestModel1 object is the results table.
Can you write Python code for it?

forest1 = tbl.Foresttrain()
forest1.target = 'Origin'
forest1.inputs = ['MPG_City','MPG_Highway','Type','Weight','Length','Cylinders']
forest1.nominals = ['Type','Cylinders']
forest1.casout = conn.CASTable('forestModel1', replace=True)
forest1()


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Tip: ExecDirect is a CAS action in the FedSQL action set to execute SQL statements in the CAS environment.
Can you write Python code for it?

conn.fedsql.execDirect(query='''
    select model,
           msrp,
           case when msrp > 80000 then 'Very Expensive' else 'Less Expensive' end as CategorySQL
    from cars_temp
    where EngineSize > 5
    order by msrp desc
''')


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Tip: Managing Your CAS Tables
How many CAS Tables have been created?
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Tip: The object "conn" created above is referred to as a CAS Connection object.
The session name is simply a human-readable tag given to a session so it is easier to identify in actions that display session information. The session ID is a unique identifier given to each session. The session ID can be used to reconnect to a session, or connect from another client.
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
To score the training data or the holdout data using the forest model, you can use the forestscore action.
Can you write Python code for it?

scored_data = conn.CASTable('scored_output', replace=True)
tbl.forestscore(modeltable=conn.CASTable('forestModel1'),casout=scored_data)
scored_data.head()


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
Using CAS' help Function
There are many actions that are available in all CAS installations, and various others may be available depending on which products have been licensed.
To display all available actions, you use the help action.
Can you write Python code for it?


I am working on the file "Python-integration/Python_Viya_Workshop.ipynb".
The first task is:
You can ask for help on specific actions and action sets.
Can you write Python code for it?


I am working on the file "Python-integration/The Data Science Pilot Action Set.ipynb".
The first task is:
Conclusion
The dataSciencePilot action set consists of actions that implement a policy-based, configurable, and scalable approach to automating data science workflows. This action set can be used to automate and end-to-end workflow or to automate steps in the  workflow such as data preparation, feature preprocessing, feature engineering, feature selection, and hyperparameter tuning.  In this notebook, we demonstrated how to use each step of the dataSciencePilot Action set using a Python interface.
Can you write Python code for it?


I am working on the file "Python-integration/The Data Science Pilot Action Set.ipynb".
The first task is:
Data Science Automated Machine Learning Pipeline
The dsAutoMl action creates a policy-based, scalable, end-to-end automated machine learning pipeline for both regression and classification problems. The only input required from the user is the input data set and the target variable, but optional parameters include the policy parameters for data exploration, variable screening, feature selection, and feature transformation.  Overriding the default policy parameters allow a data scientist to configure their pipeline in their data science workflow. In addition, a data scientist may also select additional models to consider. By default, only a decision tree model is included in the pipeline, but neural networks, random forest models, and gradient boosting models are also available.
The dsAutoMl action first explores the data and groups the input variables into categories with the same statistical profile, like the exploreData action. Next the dsAutoMl action screens variables to identify noise variables to exclude from further analysis, like the screenVariables action.  Then, the dsAutoMl action generates several new features for the input variables, like the featureMachine action. After there are various new cleaned features, the dsAutoMl action will select features based on selected criterion, like the selectFeatures action.
From here, various pipelines are created using subsets of the selected features, chosen for each pipeline using a feature-representation algorithm. Then the chosen models are added to each pipeline and the hyperparameters for the selected models are optimized, like the modelComposer action of the Autotune action set. These hyperparameters are optimized for the selected objective parameter when cross-validated. By default, classification problems are optimized to have the smallest Misclassification Error Rate (MCE) and regression problems are optimized to have the smallest Average Square Error (ASR).  Data scientists can then select their champion and challenger models from the pipelines....
Can you write Python code for it?

conn.dataSciencePilot.dsAutoMl(
    table = tbl,
    target = trt,
    explorationPolicy = expo,
    screenPolicy = scpo,
    selectionPolicy = sepo,
    transformationPolicy = trpo,
     modelTypes              = ["decisionTree", "gradboost"],
        objective               = "ASE",
        sampleSize              = 10,
        topKPipelines           = 10,
        kFolds                  = 5,
        transformationOut       = {"name" : "TRANSFORMATION_OUT_PY", "replace" : True},
        featureOut              = {"name" : "FEATURE_OUT_PY", "replace" : True},
        pipelineOut             = {"name" : "PIPELINE_OUT_PY", "replace" : True},
        saveState               = {"modelNamePrefix" : "ASTORE_OUT_PY", "replace" : True, "topK":1}
)

conn.fetch(table = {"name" : "TRANSFORMATION_OUT_PY"})

conn.fetch(table = {"name" : "FEATURE_OUT_PY"})

conn.fetch(table = {"name" : "PIPELINE_OUT_PY"})


I am working on the file "Python-integration/The Data Science Pilot Action Set.ipynb".
The first task is:
Detect Interactions
The detectInteractions action will assess the interactions between pairs of predictor variables and the correlation of that interaction on the response variable. Specially, it will see if the product of the pair of predictor variables correlate with the response variable. Since checking the correlation between the product of every predictor pair and the response variable can be computationally intensive, this action relies on the XYZ algorithm to search for these interactions efficiently in a high-dimensional space.
The detectInteractions Action requires that all predictor variables be in a binary format, but the response variable can be numeric, binary, or multi-class.  Additionally, the detectInteractions Action can handle data in a sparse format, such as when predictor variables are encoded using an one-hot-encoding scheme.  In the example below, we will specify that our inputs are sparse. The output tables shows the gamma value for each pair of variables.
Can you write Python code for it?

# Tranform data for binary format
conn.dataPreprocess.transform(
    table = hmeq,
    copyVars = ["BAD"],
    casOut = {"name": "hmeq_transform", "replace": True},
    requestPackages = [{"inputs":["JOB", "REASON"],
                        "catTrans":{"method": "label", "arguments":{"overrides":{"binMissing": True}}}},
                      {"inputs":["MORTDUE", "DEBTINC", "LOAN"],
                       "discretize": {"method": "quantile", "arguments":{"overrides":{"binMissing": True}}} }])
conn.fetch(table = {'name': 'hmeq_transform'})

conn.dataSciencePilot.detectInteractions(
    table ='hmeq_transform',
    target = trt,
    event = '1',
    sparse = True,
    inputs = ["_TR1_JOB", "_TR1_REASON", "_TR2_MORTDUE", "_TR2_DEBTINC", "_TR2_LOAN"],
    inputLevels = [7, 3, 6, 6, 6],
    casOut = {'name': 'DETECT_INT_OUT_PY', 'replace': True})
conn.fetch(table={'name':'DETECT_INT_OUT_PY'})


I am working on the file "Python-integration/The Data Science Pilot Action Set.ipynb".
The first task is:
Explore Correlations
If a target is specified, the exploreCorrelation action performs a linear and nonlinear correlation analysis of the input variables and the target. If a target is not specified, the exploreCorrelation action performs a linear and nonlinear correlation analysis for all pairwise combinations of the input variables. The correlation statistics available depend on the data type of each input variable in the pair.
- Nominal-nominal correlation pairs have the following statistics available: Mutual Information (MI), Symmetric Uncertainty (SU), Information Value (IV; for binary target), Entropy, chi-square, G test (G2), and Cramer’s V.
- Nominal-interval correlation pairs have the following statistics available: Mutual Information (MI), Symmetric Uncertainty (SU), Entropy, and F-test.
- Interval-interval correlation pairs have the following statistics available: Mutual Information (MI), Symmetric Uncertainty (SU), Entropy, and Pearson correlation.
This action returns a CAS table listing all the variable pairs and the correlation statistics.
Can you write Python code for it?

conn.dataSciencePilot.exploreCorrelation(
        table = tbl,
        casOut = {'name':'CORR_PY', 'replace':True},
        target = trt
)
conn.fetch(table = {"name" : "CORR_PY"})


I am working on the file "Python-integration/The Data Science Pilot Action Set.ipynb".
The first task is:
Feature Machine
The featureMachine action creates an automated and parallel generation of features. The featureMachine action first explores the data and groups the input variables into categories with the same statistical profile, like the exploreData action. Next the featureMachine action screens variables to identify noise variables to exclude from further analysis, like the screenVariables action.  Finally, the featureMachine action generates new features by using the available structured pipelines:
-   Missing indicator addition.
-   Mode imputation and rare value grouping.
-   Missing level and rare value grouping.
-   Median imputation.
-   Mode imputation and label encoding.
-   Missing level and label encoding.
-   Yeo-Johnson transformation and median imputation.
-   Box-Cox transformation.
-   Quantile binning with missing bins.
-   Regression tree binning.
-   Decision tree binning.
-   MDLP binning.
-   Target encoding.
-   Date, time, and datetime transformations.
Depending on the parameters specified in the transformationPolicy, the featureMachine action can generate several features for each input variable. This action returns four CAS tables: the first lists information around the transformation pipelines, the second lists information around the transformed features, the third is the input table scored with the transformed features, and the fourth is an analytical store for scoring any additional input tables.
Can you write Python code for it?

conn.dataSciencePilot.featureMachine(
    table = tbl,
    target = trt,
    copyVars = trt,
    explorationPolicy = expo,
    screenPolicy = scpo,
    transformationPolicy = trpo,
    transformationOut       = {"name" : "TRANSFORMATION_OUT", "replace" : True},
    featureOut              = {"name" : "FEATURE_OUT", "replace" : True},
    casOut                  = {"name" : "CAS_OUT", "replace" : True},
    saveState               = {"name" : "ASTORE_OUT", "replace" : True}
)

conn.fetch(table = {'name': 'TRANSFORMATION_OUT'})

conn.fetch(table = {'name': 'FEATURE_OUT'})

conn.fetch(table = {'name': 'CAS_OUT'})


I am working on the file "Python-integration/The Data Science Pilot Action Set.ipynb".
The first task is:
Generate Shadow Features
The generateShadowFeatures Action performs a scalable random permutation of input features to create shadow features. The shadow features are randomly selected from a matching distribution of each input feature. These shadow features can be used for all-relevant feature selection which removes the inputs whose variable importance is lower than the shadow feature’s variable importance. The shadow features can also be used in a post-fit analysis using Permutation Feature Importance (PFI). By replacing each input with its shadow feature one-by-one and measuring the change on model performance, one can determine that features importance based on relative size of the model’s performance change.
In the example below, I will use the outputs of the feature machine for all-relevant feature selection. This involves getting the variable metadata from my feature machine table, generating my shadow features, finding the variable importance for my features and shadow features using a random forest, and comparing each variable's performance to its shadow features. In the end, I will only keep variables with a higher importance than its shadow feature for the next phase.
Can you write Python code for it?

# Getting variable names and metadata from feature machine output
fm = conn.CASTable('FEATURE_OUT').to_frame()
inputs = fm['Name'].to_list()
nom = fm.loc[fm['IsNominal'] == 1]
nom = nom['Name'].to_list()

# Generating Shadow Features
conn.dataSciencePilot.generateShadowFeatures(
    table = 'CAS_OUT',
    nProbes = 2,
    inputs = inputs,
    nominals = nom,
    casout={"name" : "SHADOW_FEATURES_OUT", "replace" : True},
    copyVars = trt
)
conn.fetch(table = {"name" : "SHADOW_FEATURES_OUT"})

# Getting Feature Importance for Orginal Features
feats = conn.decisionTree.forestTrain(
    table = 'CAS_OUT',
    inputs = inputs,
    target = trt,
    varImp = True)
real_features = feats.DTreeVarImpInfo

# Getting Feature Importance for Shadow Features
inp = conn.CASTable('SHADOW_FEATURES_OUT').axes[1].to_list()
shadow_feats = conn.decisionTree.forestTrain(
    table = 'SHADOW_FEATURES_OUT',
    inputs = inp,
    target = trt,
    varImp = True)
sf = shadow_feats.DTreeVarImpInfo

# Building dataframe for easy comparison
feat_comp = pd.DataFrame(columns=['Variable', 'Real_Imp', 'SF_Imp1', 'SF_Imp2'])
# Filling Variable Column of Data Frame from Feature
feat_comp['Variable'] = real_features['Variable']
# Filling Importance Column of Data Frame from Feature
feat_comp['Real_Imp'] = real_features['Importance']
# Finding each Feature's Shadow Feature
for index, row in sf.iterrows():
    temp_name = row['Variable']
    temp_num = int(temp_name[-1:])
    temp_name = temp_name[5:-2]
    temp_imp = row['Importance']
    for ind, ro in feat_comp.iterrows():
        if temp_name == ro['Variable']:
            if temp_num == 1:
                # Filling First Shadow Feature's Importance
                feat_comp.at[ind, 'SF_Imp1'] = temp_imp
            else:
                # Filling First Shadow Feature's Importance
                feat_comp.at[ind, 'SF_Imp2'] = temp_imp
feat_comp.head()

# Determining which features have an importance smaller than their shadow feature's importance
to_drop = list()
for ind, ro in...


I am working on the file "Python-integration/The Data Science Pilot Action Set.ipynb".
The first task is:
Select Features
The selectFeatures action performs a filter-based selection by the criterion selected in the selectionPolicy (default is the best ten input variables according to the Mutual Information statistic). The criterion available for selection include Chi-Square, Cramer’s V, F-test, G2, Information Value, Mutual Information, Normalized Mutual Information statistic, Pearson correlation, and the Symmetric Uncertainty statistic. This action returns a CAS table listing the variables, their rank according to the selected criterion, and the value of the selected criterion.
Can you write Python code for it?

conn.dataSciencePilot.screenVariables(
    table='CAS_OUT',
    target=trt,
    screenPolicy=scpo,
    casout={"name" : "SCREEN_VARIABLES_OUT", "replace" : True}
)
conn.fetch(table = {"name" : "SCREEN_VARIABLES_OUT"})


I am working on the file "Python-integration/Viya 2020 Example.ipynb".
The first task is:
Accessing Models via API
Now, we can head over to SAS Viya to manage our model using the visual interface. But, you don't need to use the visual interface in SAS Viya if you don't want to! You can actually access most of the model management capabilities through the SAS open source package SASCTL. For whatever isn't included in the open source package, you can do through API!
In this section, we will use the SAS Viya APIs to look at one of our deployed models. To use this notebook on your own, make sure you have deployed one of your models, either through the visual interface or programmatically. To get started, we need to generate an authorization token.
Can you write Python code for it?

from requests import request
import urllib3
urllib3.disable_warnings()

# Get token
host = 'http://' + hostname
url = host + '/SASLogon/oauth/token'
r = request('POST', url,
            data='grant_type=password&username=%s&password=%s' %(username, password),
            headers={
                'Accept': 'application/json',
                'Content-Type': 'application/x-www-form-urlencoded'
            },
            auth=('sas.ec', ''),
            verify=False)
token = r.json()['access_token']


I am working on the file "Python-integration/Viya 2020 Example.ipynb".
The first task is:
Data Science Pilot in Python
Now we will interface with the Data Science Pilot Action Set to build the same model we can build in SAS in Python - demonstrating the ability to code in both Python and SAS, while getting the same results!
Can you write Python code for it?

conn.builtins.loadactionset('dataSciencePilot')
conn.dataSciencePilot.dsAutoMl(
    table = 'hmeqct',
    target = 'BAD',
    explorationPolicy = {'cardinality': {'lowMediumCutoff':40}},
    screenPolicy = {'missingPercentThreshold':35},
    selectionPolicy = {'topk':10},
    transformationPolicy = {'entropy': True, 'iqv': True, 'kurtosis': True, 'outlier': True},
    modelTypes              = ["decisionTree", "gradboost"],
    objective               = "AUC",
    sampleSize              = 10,
    topKPipelines           = 5,
    kFolds                  = 2,
    transformationOut       = {"name" : "TRANSFORMATION_OUT_PY", "replace" : True},
    featureOut              = {"name" : "FEATURE_OUT_PY", "replace" : True},
    pipelineOut             = {"name" : "PIPELINE_OUT_PY", "replace" : True},
    saveState               = {"modelNamePrefix" : "ASTORE_OUT_PY", "replace" : True, "topK":1}
)


I am working on the file "Python-integration/Viya 2020 Example.ipynb".
The first task is:
Great! Now let's use our feature creation model and our grad boost model to score our data.
Can you write Python code for it?

conn.loadactionset('astore')
model_astore = conn.CASTable("ASTORE_OUT_PY_gradBoost_1")
feature_astore = conn.CASTable("ASTORE_OUT_PY_fm_")
conn.score(
    table='hmeqct',
    copyvars='BAD',
    out = dict(name='feat_scored', replace=True),
    rstore='ASTORE_OUT_PY_fm_')
conn.score(
    table='feat_scored',
    copyvars='BAD',
    out = dict(name='hmeq_scored', replace=True),
    rstore='ASTORE_OUT_PY_gradBoost_1')
hmeq_scored = conn.CASTable('hmeq_scored')
hmeq_scored = hmeq_scored[['BAD', 'P_BAD1']].to_frame()
hmeq_scored.BAD = hmeq_scored.BAD.astype(int)
hmeq_scored = hmeq_scored.rename(columns={'BAD': '0', 'P_BAD1': '0'})
hmeq_scored = pd.DataFrame(hmeq_scored)


I am working on the file "Python-integration/Viya 2020 Example.ipynb".
The first task is:
Let's pull the specific information for our deployed module.
Can you write Python code for it?

deployed_name= input("Deployed Name:")

headers = {'Authorization': 'Bearer ' + token}

url = host + '/microanalyticScore/modules/' + deployed_name + '/steps'

r = request('GET', url, params={}, headers = headers, verify=False)

r.json()