I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
same distribution, different mean (between success sample and fail sample)

H. Funding_duration
Can you write Python code for it?

plt.figure(figsize=(10,8));
sns.kdeplot(cf_df.loc[cf_df['end_with_success'] ==True]['funding_duration'], label = 'success');
sns.kdeplot(cf_df.loc[cf_df['end_with_success'] ==False]['funding_duration'], label = 'fail');
plt.yticks(fontsize=15);
plt.xlabel('Funding_duration', fontsize=15);
plt.ylabel('Distribution', fontsize = 15);
plt.legend(fontsize = 15);

# Ks_2sampResult : Kolmogorov-Smirnov test
# Ttest_indResult : 2 sample T-test
success_duration = cf_df.loc[cf_df['end_with_success'] ==True]['funding_duration']
fail_duration = cf_df.loc[cf_df['end_with_success'] ==False]['funding_duration']
print(sp.stats.ks_2samp(success_duration, fail_duration))
print(sp.stats.ttest_ind(success_duration, fail_duration))


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
when using k=4, (target_money, funding_duration, start_month, has_video) are selected

C. Correlation
Can you write Python code for it?

x_categories = pd.DataFrame([x['category_label'], x['has_video'], x['start_month']]).T
x_nominal = pd.DataFrame([x['target_money'], x['grammar_level'], x['funding_duration'],
                         x['num_funding_type']]).T

category_corr = x_categories.corr(method='spearman')
print(category_corr);
sns.heatmap(category_corr);
plt.xticks(fontsize = 15);
plt.yticks(fontsize = 15);

nominal_corr = x_nominal.corr(method='pearson')
print(nominal_corr)
sns.heatmap(nominal_corr);
plt.xticks(fontsize = 15, rotation='vertical');
plt.yticks(fontsize = 15);


I am working on the file "1. DataCrawling + Cleansing.ipynb".
The first task is:
3. Project_money Data 처리

날짜별 펀딩된 금액 중 0~5일 內 펀딩금액만 처리
Can you write Python code for it?

wadiz_df['project_id'] = wadiz_df['project_id'].apply(lambda x: int(x))

# 날짜 계산용 DataFrame 생성
date_difference = pd.merge(project_money_all, wadiz_df, on="project_id")

# funding_date 처리
project_money_all["funding_date"] = pd.to_datetime(project_money_all["funding_date"])
date_difference["funding_date"] = pd.to_datetime(date_difference["funding_date"])
date_difference["date_start"] = pd.to_datetime(date_difference["date_start"])

date_difference["funding_date"] - date_difference["date_start"]

# 프로젝트 개설일과 개인별 펀딩일 차이
project_money_all["date_difference"] = date_difference["funding_date"] - date_difference["date_start"]
# NaN값 제거 (이상치)
project_money_all['date_difference'] = project_money_all['date_difference'].fillna('-1')
project_money_all = project_money_all[project_money_all['date_difference'] >= '0 days']

# 날짜 처리
project_money_all["date_difference"] = project_money_all["date_difference"].apply(lambda x: int(x)/8.640000e+13)
#project_money_all = project_money_all[project_money_all["date_difference"] >= 0]

project_money_all

type(project_money_all['date_difference'][0])

project_money_all["0day_difference"] = np.ones(len(project_money_all))

# 0~5일 이내 funding된 금액만 처리
for i in np.arange(6):
    number = i
    project_money_all["{number}day_difference".format(number = i)] = np.ones(len(project_money_all))
    project_money_all["{number}day_difference".format(number = i)][project_money_all["date_difference"] <= number] = "short"
    project_money_all["{number}day_difference".format(number = i)][project_money_all["date_difference"] > number] = "long"

project_money_all['funding_money'] = project_money_all['funding_money'].apply(lambda x: int(x))

zero_day = project_money_all.loc[project_money_all["0day_difference"] == "short"]
one_day = project_money_all.loc[project_money_all["1day_difference"] == "short"]
two_day = project_money_all.loc[project_money_all["2day_difference"] == "short"]
three_day = project_money_all.loc[project_money_all["3day_difference"] == "short"]
four_day =...


I am working on the file "1. DataCrawling + Cleansing.ipynb".
The first task is:
4. Project_money_all, Wadiz_df 합치기

각 프로젝트별 0~5일 內 누적금액이 얼마인지 확인가능
Can you write Python code for it?

wadiz_df = pd.merge(wadiz_df, zero_day, on = "project_id", how='outer')
wadiz_df = pd.merge(wadiz_df, one_day, on = "project_id", how='outer')
wadiz_df = pd.merge(wadiz_df, two_day, on = "project_id", how='outer')
wadiz_df = pd.merge(wadiz_df, three_day, on = "project_id", how='outer')
wadiz_df = pd.merge(wadiz_df, four_day, on = "project_id", how='outer')
wadiz_df = pd.merge(wadiz_df, five_day, on = "project_id", how='outer')


I am working on the file "1. DataCrawling + Cleansing.ipynb".
The first task is:
5. Comment Crawling

각 프로젝트별 지지자, 개설자 댓글 크롤링
user_data_all : 지지자 댓글
provider_data_all : 개설자 댓글
Can you write Python code for it?

project_id = wadiz_df.project_id
user_data = pd.DataFrame(columns=['project_id', 'user_id', 'comment', 'date'])
user_data_all = pd.DataFrame()
provider_data = pd.DataFrame(columns=['project_id', 'provider_id', 'comment', 'date'])
provider_data_all = pd.DataFrame()
for i in project_id[0:]:
    project_id_list = i
    response = requests.get('https://www.wadiz.kr/web/campaign/detail/qa/{project_id_list}'.format(project_id_list = i))
    dom = BeautifulSoup(response.content, 'html.parser')
    user_all = dom.select('div.wd-ui-recommend li.')
    print(project_id_list)
    if len(user_all) == 0:
        pass
    else:
        for number in np.arange(len(user_all)):
            user = user_all[number]
            user_url = user.select_one('a.wd-data-name').get('href')
            user_comment = user.select_one('span').text
            try:
                user_date = user.select_one('span.wd-data-whenCreated').text
            except:
                continue
            user_data.loc[len(user_data)] = [project_id_list, user_url, user_comment, user_date]
    provider_all = dom.select('ul.wd-list-reply')
    #print(project_id_list)
    for number in np.arange(len(provider_all)):
        provider = provider_all[number]
        provider_url = provider.select_one('a.wd-data-name').get('href')
        provider_comment = provider.select('span')[-2].text
        provider_date = provider.select('span')[-1].text
        provider_data.loc[len(provider_data)] = [project_id_list, provider_url, provider_comment, provider_date]
user_data_all = user_data_all.append(user_data)
provider_data_all = provider_data_all.append(provider_data)

user_data_all.to_csv('user_data_all_0329.csv', encoding='utf-8')
provider_data_all.to_csv('provider_data_all_0329.csv', encoding='utf-8')


I am working on the file "1. DataCrawling + Cleansing.ipynb".
The first task is:
5. Data 추가
Can you write Python code for it?

# NaN 값 체크
# Nan값은 0~5일내에 펀딩된 금액이 없는 것을 뜻함
for i in wadiz_df.columns:
    column = i
    print(len(wadiz_df.loc[wadiz_df["{column}".format(column = i)].isnull() == True]))

wadiz_df.fillna(0, inplace=True)

#NaN 값 다시 체크
for i in wadiz_df.columns:
    column = i
    print(len(wadiz_df.loc[wadiz_df["{column}".format(column = i)].isnull() == True]))

#funding_rate 생성
for i in np.arange(6):
    number = i
    wadiz_df["{number}day_funding_rate".format(number = i)] = \
    wadiz_df["{number}day_funding_money".format(number = i)]/wadiz_df["target"]

# funding_rate 1 이상인 값들 체크
for i in np.arange(6):
    number = i
    print(len(wadiz_df.loc[wadiz_df["{number}day_funding_rate".format(number = i)] >= 1]))

# funding_rate -> log scale
for i in np.arange(6):
    number = i
    wadiz_df["{number}day_log_funding_rate".format(number = i)] = wadiz_df["{number}day_funding_rate"\
                                                                           .format(number = i)].apply(lambda x: np.log(x))

wadiz_df.to_csv('wadiz_df_0329_1.csv', encoding='utf-8')


I am working on the file "1. DataCrawling + Cleansing.ipynb".
The first task is:
6. 맞춤법 검사

한글 맞춤법 검사기 (python 2.7, https://github.com/ssut/py-hanspell)

개설자 댓글 맞춤법 검사
error 갯수 합산, 각 댓글별 어절 수 계산
provider_grammar_level : error / comment_length(어절 수)
Can you write Python code for it?

comment_analysis = pd.DataFrame(columns={'project_id', 'provider_id',
                         'result', 'original', 'checked', 'words', 'time', 'comment_length'})
for i in np.arange(len(provider_data_all)):
    try:
        result = spell_checker.check(provider_data_all['comment'][i])
        comment = pd.DataFrame(provider_data_all.loc[i]).T
        comment_result = pd.DataFrame([result])
        comment_result.index = comment.index
        comment_result_df = comment.join(comment_result)
        comment_analysis = comment_analysis.append(comment_result_df)
        comment_analysis['comment_length'][i] = len(comment_analysis['words'][i])
        if i in 100*np.arange(220):
            print(i)
    except:
        continue

# comment_error 생성
comment_error = pd.DataFrame([comment_analysis.project_id, comment_analysis.errors,
                              comment_analysis.provider_id, comment_analysis.comment_length]).T
# data int타입으로 전환
comment_error['errors'] = comment_error['errors'].apply(lambda x: int(x))
comment_error['comment_length'] = comment_error['comment_length'].apply(lambda x: int(x))

# comment error 처리
comment_error['errors'] = comment_error['errors'].apply(lambda x: int(x))
comment_error['comment_length'] = comment_error['comment_length'].apply(lambda x: int(x))

# id로 groupby
comment_error = comment_error.groupby(by='project_id', as_index=False).sum()

# grammar_level 생성
# 각 댓글에 속한 error를 전체 어절로 나눔
comment_error['provider_grammar_level'] = comment_error['errors']/comment_error['comment_length']

comment_analysis.to_csv('comment_analysis.csv', encoding='utf-8')


I am working on the file "1. DataCrawling + Cleansing.ipynb".
The first task is:
7. 개설자 댓글이 존재하는 Project만 선정

wadiz_provider_analysis : 총 샘플 수 599개 (
Can you write Python code for it?

wadiz_provider_analysis = pd.merge(wadiz_df, comment_error, how='inner', on= 'project_id')

wadiz_provider_analysis.to_csv('wadiz_provider_analysis_0329.csv', encoding='utf-8')

wadiz_df.head()

wadiz_provider_analysis

project_money_all

provider_data_all

user_data_all


I am working on the file "1. Examine calibration - SMI 60 Hz.ipynb".
The first task is:
A "dataframe is just a large spreadsheet.  Lets have a look! at the first ten rows of data.
Can you write Python code for it?


I am working on the file "1. Examine calibration - SMI 60 Hz.ipynb".
The first task is:
According to our figures above, we need to flip the X axis of the SMI data to match the Vizard coordinate system.  We can actually do this for all the cyclopean gaze data in one line of code by applying an anonymous function to each row of the dataframe.
The anonymous function below iterates over the rows of the dataframe, and the row's values are stored in the variable that I chose to call "row."
Can you write Python code for it?

dataFrame['cycEyeInHead_XYZ'] = dataFrame['cycEyeInHead_XYZ'].apply(lambda row: [-row[0],row[1],row[2]])


I am working on the file "1. Examine calibration - SMI 60 Hz.ipynb".
The first task is:
Assign values to columns of the dataframe
Can you write Python code for it?

dataFrame['eihAzimuthDeg'] = eihAzimuthDeg
dataFrame['eihElevationDeg'] = eihElevationDeg

dataFrame['calibAzimuthDeg'] = calibAzimuthDeg
dataFrame['calibElevationDeg'] = calibElevationDeg

dataFrame['calErr2D'] = calErr2D


I am working on the file "1. Examine calibration - SMI 60 Hz.ipynb".
The first task is:
Great!  Now all of our data is in the Vizard coordinate system.  That makes things much easier to deal with, going forward.
Plot calibration data
Lets have a look at our eye tracker's calibration.  I'll walk you through this, and will provide some convenience functions along the way.
First, we need to organize the data.
During calibraiton, we had subjects fixate a series of calibration points.  The location fo the current cabliration point is stored in each row of the dataframe.  While the subject was fixating a cross, we pressed a button that labelled the next 200 ms with a unique "trialNumber."  The trial number corresponds to which point they were looking at, starting at trialNumber 1000 for the top-left point.  This facilitates grabbing all the frames on which the subject was looking at that point.  Lets grab those frames!
First, we can tell Pandas that we want to group our rows by the variable 'trialNumber'
Can you write Python code for it?


I am working on the file "1. Examine calibration - SMI 60 Hz.ipynb".
The first task is:
Import the data into a "Pandas" dataFrame.
Pandas is a Python module for handling large data structres like this.
Can you write Python code for it?


I am working on the file "1. Examine calibration - SMI 60 Hz.ipynb".
The first task is:
It will return an error for certain rows.  Can't avoid that!
Can you write Python code for it?

out = dataFrame.apply( lambda row: calculateGazeError(row),axis=1)
(calErr2D,eihAzimuthDeg,eihElevationDeg,calibAzimuthDeg,calibElevationDeg) = zip(*out)


I am working on the file "1. Examine calibration - SMI 60 Hz.ipynb".
The first task is:
Lets have another look and see if that worked.  Was the X axis flipped?
Can you write Python code for it?


I am working on the file "1. Examine calibration - SMI 60 Hz.ipynb".
The first task is:
Now, lets take the mean and standard deviation using the "aggregate function"
Can you write Python code for it?

gbTrial = dataFrame.groupby('trialNumber')
calibStatsDF = gbTrial.aggregate([np.mean,np.std])


I am working on the file "1. Examine calibration - SMI 60 Hz.ipynb".
The first task is:
The variable "gbTrial" is not a dataframe, but a series of pointers to each "group," where a group is a collection of rows  with the same trial number.  If you want to have a look, uncomment the next line of code.  It returns a lot of text, so I'm leaving it commented to begin with.
Can you write Python code for it?


I am working on the file "1. Feed-forward Deep Neural Networks.ipynb".
The first task is:
Convolution Networks:
The following code defines our convolution network and trains it using a GPU/CPU.
Can you write Python code for it?

X_train, y_train, X_val, y_val, X_test, y_test = load_dataset()

net1 = NeuralNet(
    layers=[('input', layers.InputLayer),
            ('conv2d1', layers.Conv2DLayer),
            ('maxpool1', layers.MaxPool2DLayer),
            ('conv2d2', layers.Conv2DLayer),
            ('maxpool2', layers.MaxPool2DLayer),
            ('dropout1', layers.DropoutLayer),
            ('dense', layers.DenseLayer),
            ('dropout2', layers.DropoutLayer),
            ('output', layers.DenseLayer),
            ],
    # input layer
    input_shape=(None, 1, 28, 28),
    # layer conv2d1
    conv2d1_num_filters=16,
    conv2d1_filter_size=(6, 6),
    conv2d1_nonlinearity=lasagne.nonlinearities.rectify,
    conv2d1_W=lasagne.init.GlorotUniform(),
    # layer maxpool1
    maxpool1_pool_size=(2, 2),
    # layer conv2d2
    conv2d2_num_filters=16,
    conv2d2_filter_size=(5, 5),
    conv2d2_nonlinearity=lasagne.nonlinearities.rectify,
    # layer maxpool2
    maxpool2_pool_size=(2, 2),
    # dropout1
    dropout1_p=0.5,
    # dense
    dense_num_units=192,
    dense_nonlinearity=lasagne.nonlinearities.rectify,
    # dropout2
    dropout2_p=0.5,
    # output
    output_nonlinearity=lasagne.nonlinearities.softmax,
    output_num_units=10,
    # optimization method params
    update=nesterov_momentum,
    update_learning_rate=0.01,
    update_momentum=0.9,
    max_epochs=8,
    verbose=1,
)

# Train the network
nn = net1.fit(X_train, y_train)


I am working on the file "1. Feed-forward Deep Neural Networks.ipynb".
The first task is:
MNIST Dataset:
The MNIST dataset is the common first dataset used as a demo in deep learning. More specifically, it's a digit classification learning model.
Can you write Python code for it?

def load_dataset():
    with gzip.open("datasets/mnist/mnist.pkl.gz") as fin:
        data = pickle.load(fin)

    X_train, y_train = data[0]
    X_val, y_val = data[1]
    X_test, y_test = data[2]

    X_train = X_train.reshape((-1, 1, 28, 28))
    X_val = X_val.reshape((-1, 1, 28, 28))
    X_test = X_test.reshape((-1, 1, 28, 28))

    y_train = y_train.astype(np.uint8)
    y_val = y_val.astype(np.uint8)
    y_test = y_test.astype(np.uint8)

    return X_train, y_train, X_val, y_val, X_test, y_test


I am working on the file "1. Feed-forward Deep Neural Networks.ipynb".
The first task is:
Prediction:
Can you write Python code for it?


I am working on the file "1. Feed-forward Deep Neural Networks.ipynb".
The first task is:
Theano & Feature Extraction:
Can you write Python code for it?

dense_layer = layers.get_output(net1.layers_['dense'], deterministic=True)
output_layer = layers.get_output(net1.layers_['output'], deterministic=True)
input_var = net1.layers_['input'].input_var

f_output = theano.function([input_var], output_layer)
f_dense = theano.function([input_var], dense_layer)


I am working on the file "1. First steps with Python.ipynb".
The first task is:
"What on earth?!" you might ask in incredulity. "Python clearly fails math."
You have just attempted to divide one integer by another. And, as it happens, when you math in Python you will get out what you get in. In this case: an integer (or, in Python, an int).
Of course, we want to deal with all kinds of other real numbers, like 5.1 and $10^-5$. For this we have the floating-point numbers, or float. To write a float, just include a decimal (.):
Can you write Python code for it?


I am working on the file "1. First steps with Python.ipynb".
The first task is:
(if it looks weird to put the function at the end of the string, don't worry; we'll cover that later)
We can get the third character in the string:
Can you write Python code for it?

mycoolstring = "I'm sick of writing strings, so I'll define a variable."

mycoolstring[2]    # The third character.


I am working on the file "1. First steps with Python.ipynb".
The first task is:
...and subtract like this:
Can you write Python code for it?


I am working on the file "1. First steps with Python.ipynb".
The first task is:
1.2. Floats and Integers
Suppose I want to divide 5 by 3. You might be tempted to do...
Can you write Python code for it?


I am working on the file "1. First steps with Python.ipynb".
The first task is:
But they have to match:
Can you write Python code for it?


I am working on the file "1. First steps with Python.ipynb".
The first task is:
First I calculated the area of a circle, and assigned it to a variable called area_circle. Then I calculated the area of the cylinder, using area_circle and the other two parameters (radius and height).
Can you write Python code for it?


I am working on the file "1. First steps with Python.ipynb".
The first task is:
Just like with strings, I can retrieve items at specific positions in a list.
Can you write Python code for it?

widgets[1]    # Remember, the --second-- item.


I am working on the file "1. First steps with Python.ipynb".
The first task is:
Notice this thing we're calling the parameter, radius. radius is a variable. What we've said is that when we call the function, like we just did, whatever we provide as the first (and only) input value should be assigned to a variable called radius. Inside the function, we can use that variable however we like.
We can write as many procedures as we like inside of our function. Suppose, for example, that we wanted to calculate the area of a right cylinder. Recall that:
$A_{cylinder} = 2\pi rh + A_{circle}$
Where $h$ is the height of the cylinder on the axis perpendicular to the radius.
So we could define a function like so:
Can you write Python code for it?

def area_of_a_cylinder(radius, height):
    area_circle = area_of_a_circle(radius)
    return 2. * 3.1415927 * radius * height + area_circle


I am working on the file "1. First steps with Python.ipynb".
The first task is:
Oh, and the last item!
Can you write Python code for it?

widgets[-1]    # -2 would give the second from last.


I am working on the file "1. First steps with Python.ipynb".
The first task is:
The first line,
python
def area_of_a_circle(radius):
is the declaration or signature of the function. This part says, "I hereby define (def) a function called area_of_a_circle. It shall accept a single parameter, which I shall call radius. Let it be so! (:)"
The second line,
python
return 3.1415927 * radius ** 2
Tells Python what your so-declared function should return. In this case, it returns the outcome of the mathematical operation 3.1415927 * radius ** 2.
Let's try it:
Can you write Python code for it?


I am working on the file "1. First steps with Python.ipynb".
The first task is:
Want to calculate the remainder of a division operation? Use the modulo (%) operator!
Can you write Python code for it?

5 % 2    # 5 divided by 2 is 2, with a remainder of 1!


I am working on the file "1. First steps with Python.ipynb".
The first task is:
We can also make strings upper or lower case, or even title case! In an IPython notebook, only the value of the last expression is displayed. So if we want to see the value of multiple expressions in the same cell we'll have to print them.
Can you write Python code for it?

print 'WHY AM I SCREAMINg?!'.lower()
print "Often this is useful for getting someone's attention".upper()
print 'french people often forget to capitalize proper nouns'.title()


I am working on the file "1. First steps with Python.ipynb".
The first task is:
You can also use single-quotes:
Can you write Python code for it?


I am working on the file "1. First steps with Python.ipynb".
The first task is:
You can concatenate strings using the plus (+) operator.
Can you write Python code for it?

'I am the first part of the string, ' + 'and I am the second part.'


I am working on the file "1. First steps with Python.ipynb".
The first task is:
You can perform multiple operations in a single statement. Python will evaluate your multi-operation statement in the following order:

Power. x ** y
Multiplication, division, modulo. x * y, x / y, x % y
Addition, subtraction. x + y, x - y
Can you write Python code for it?


I am working on the file "1. Intro to Jupyter Notebooks - Python.ipynb".
The first task is:
Example 2: Video Game Sales
Can you write Python code for it?

games = pd.read_csv("videogames.csv", sep = ",")

games.head()

by_publisher = games.groupby("Publisher").agg({"NA_Sales": sum,
                                               "EU_Sales": sum,
                                               "JP_Sales": sum,
                                               "Global_Sales": sum,
                                               "Critic_Score": np.mean})
by_publisher["Nintendo"]

by_publisher.columns

top_publishers = by_publisher.sort_values("Global_Sales", ascending = False)[0:15][["NA_Sales", "EU_Sales", "JP_Sales"]]
top_publishers

top_publishers.plot(kind="bar", figsize=(12,5))

# And again, as a barplot
top_publishers.plot(kind="bar", stacked = True, figsize=(12,5))


I am working on the file "1. Iterations/Binary_Gap.ipynb".
The first task is:
n = 10
Can you write Python code for it?

n = 10

print n.bit_length()

a = n.bit_length()

print bin(n)

print '%0*d' % (a, int(bin(n)[2:]))

print '{0:08b}'.format(n)


I am working on the file "1. Iterations/Binary_Gap.ipynb".
The first task is:
n=100
Can you write Python code for it?


I am working on the file "1. Iterations/Binary_Gap.ipynb".
The first task is:
testing more binary to decimal conversions
Can you write Python code for it?

n = 37

count_one = 0
count_gap = 0
binarygap = 0

for count in xrange(n.bit_length()):
    print count
    print "n", n

    if n % 2:
        count_one +=1
        n = n /2
    elif count_one == 1:
        count_gap += 1
        print "count gap: ", count_gap
        n = n / 2

    if count_one > 1:
        if binarygap < count_gap:
            binarygap = count_gap
            print "binary gap", binarygap
        count_one = 1
        count_gap = 0

print binarygap

n = 142

count_one = 0
count_gap = 0
binarygap = 0

for count in xrange(n.bit_length()):
    print count
    print "n", n

    if n % 2:
        count_one +=1
        n = n /2
    elif count_one == 1:
        count_gap += 1
        print "count gap: ", count_gap
        n = n / 2
    else:
        n = n/2

    if count_one > 1:
        if binarygap < count_gap:
            binarygap = count_gap
            print "binary gap", binarygap
        count_one = 1
        count_gap = 0

print binarygap

142 / 2

71/2

35/2

n = 488

count_one = 0
count_gap = 0
binarygap = 0

for count in xrange(n.bit_length()):
    print count
    print "n", n

    if n % 2:
        count_one +=1
        n = n /2
    elif count_one == 1:
        count_gap += 1
        print "count gap: ", count_gap
        n = n / 2
    else:
        n = n/2

    if count_one > 1:
        if binarygap < count_gap:
            binarygap = count_gap
            print "binary gap", binarygap
        count_one = 1
        count_gap = 0

print binarygap

n = 181

count_one = 0
count_gap = 0
binarygap = 0

for count in xrange(n.bit_length()):
    print count
    print "n", n

    if n % 2:
        count_one +=1
        n = n /2
    elif count_one == 1:
        count_gap += 1
        print "count gap: ", count_gap
        n = n / 2
    else:
        n = n/2

    if count_one > 1:
        if binarygap < count_gap:
            binarygap = count_gap
            print "binary gap", binarygap
        count_one = 1
        count_gap = 0

print binarygap

def...


I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
Basic Concepts
What is "learning from data"?

In general Learning from Data is a scientific discipline that is concerned with the design and development of algorithms that allow computers to infer (from data) a  model that allows compact representation (unsupervised learning) and/or good generalization (supervised learning).

This is an important technology because it enables computational systems to adaptively improve their performance with experience accumulated from the observed data.
Most of these algorithms are based on the iterative solution of a mathematical problem that involves data and model. If there was an analytical solution to the problem, this should be the adopted one, but this is not the case for most of the cases.
So, the most common strategy for learning from data is based on solving a system of equations as a way to find a series of parameters of the model that minimizes a mathematical problem. This is called optimization.
The most important technique for solving optimization problems is gradient descend.
Preliminary: Nelder-Mead method for function minimization.
The most simple thing we can try to minimize a function $f(x)$ would be to sample two points relatively near each other, and just repeatedly take a step down away from the largest value. This simple algorithm has a severe limitation: it can't get closer to the true minima than the step size.
The Nelder-Mead method dynamically adjusts the step size based off the loss of the new point. If the new point is better than any previously seen value, it expands the step size to accelerate towards the bottom. Likewise if the new point is worse it contracts the step size to converge around the minima. The usual settings are to half the step size when contracting and double the step size when expanding.
This method can be easily extended into higher dimensional examples, all that's required is taking one more point than there are dimensions. Then, the simplest approach is to replace the worst point with a point reflected through the...
Can you write Python code for it?

# numerical derivative at a point x

def f(x):
    return x**2

def fin_dif(x, f, h = 0.00001):
    '''
    This method returns the derivative of f at x
    by using the finite difference method
    '''
    return (f(x+h) - f(x))/h

x = 2.0
print "{:2.4f}".format(fin_dif(x,f))


I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
Choosing Alpha
The step size, alpha, is a slippy concept: if it is too small we will slowly converge to the solution, if it is too large we can diverge from the solution.
There are several policies to follow when selecting the step size:

Constant size steps. In this case, the size step determines the precision of the solution.
Decreasing step sizes.
At each step, select the optimal step (the one that get the lower $f(\mathbf x)$).

The last policy is good, but too expensive. In this case we would consider a fixed set of values:
Can you write Python code for it?


I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
Example: Stochastic Gradient Descent and Linear Regression
The linear regression model assumes a linear relationship between data:
$$ y_i = w_1 x_i + w_0 $$
Let's generate a more realistic dataset (with noise), where $w_1 = 2$ and $w_0 = 0$.

The bias trick. It is a little cumbersome to keep track separetey of $w_i$, the feature weights, and $w_0$, the bias. A common used trick is to combine these parameters into a single structure that holds both of them by extending the vector $x$ with one additional dimension that always holds the constant $1$. With this dimension the model simplifies to a single multiply $f(\mathbf{x},\mathbf{w}) = \mathbf{w} \cdot \mathbf{x}$.
Can you write Python code for it?

%reset
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline

# x: input data
# y: noisy output data

x = np.random.uniform(0,1,20)

# f = 2x + 0
def f(x): return 2*x + 0

noise_variance =0.1
noise = np.random.randn(x.shape[0])*noise_variance
y = f(x) + noise

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.xlabel('$x$', fontsize=15)
plt.ylabel('$f(x)$', fontsize=15)
plt.plot(x, y, 'o', label='y')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)')
plt.ylim([0,2])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show

# f_target = 1/n Sum (y - wx)**2
def target_f(x,y,w):
    return np.sum((y - x * w)**2.0) / x.size

# gradient_f = 2/n Sum 2wx**2 - 2xy
def gradient_f(x,y,w):
    return 2 * np.sum(2*w*(x**2) - 2*x*y) / x.size

def in_random_order(data):
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

def SGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        epochs=100,
        alpha_0=0.01):

    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    iteration_no_increase = 0
    w_cost = []
    epoch = 0
    while epoch < epochs and iteration_no_increase < 100:
        val = target_f(x, y, w)
        if min_val - val > toler:
            min_w, min_val = w, val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            iteration_no_increase += 1
            alpha *= 0.9
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = w - (alpha *  gradient_i)
            w_cost.append(target_f(x,y,w))
        epoch += 1
    return min_w, np.array(w_cost)

w, target_value = SGD(target_f, gradient_f, x, y)
print 'w: {:.6f}'.format(w)

fig, ax = plt.subplots(1,...


I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
From derivatives to gradient: $n$-dimensional function minimization.
Let's consider a $n$-dimensional function $f: \Re^n \rightarrow \Re$. For example:
$$f(\mathbf{x}) = \sum_{n} x_n^2$$
Our objective is to find the argument  $\mathbf{x}$ that minimizes this function.
The gradient of $f$ is the vector whose components are the $n$ partial derivatives of $f$. It is thus a vector-valued function.
The gradient points in the direction of the greatest rate of increase of the function.
$$\nabla {f} = (\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n})$$
Can you write Python code for it?

def f(x):
    return sum(x_i**2 for x_i in x)

def fin_dif_partial_centered(x, f, i, h=1e-6):
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    w2 = [x_j - (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(w2))/(2*h)

def gradient_centered(x, f, h=1e-6):
    return[round(fin_dif_partial_centered(x,f,i,h), 10)
           for i,_ in enumerate(x)]

x = [1.0,1.0,1.0]

print '{:.6f}'.format(f(x)), gradient_centered(x,f)


I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
It can be shown that the “centered difference formula" is better when computing numerical derivatives:
$$ \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h} $$
The error in the "finite difference" approximation can be derived from Taylor's theorem and, assuming that $f$ is differentiable, is $O(h)$. In the case of “centered difference" the error is $O(h^2)$.
The derivative tells how to chage $x$ in order to make a small improvement in $f$.
Minimization
Then, we can follow these steps to decrease the value of the function:

Start from a random $x$ value.
Compute the derivative $f'(x) = \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h}$.
Walk a small step (possibly weighted by the derivative module) in the opposite direction of the derivative, because we know that $f(x - h \mbox{ sign}(f'(x))$ is less than $f(x)$ for  small enough $h$.

The search for the minima ends when the derivative is zero because we have no more information about which direction to move. $x$ is a critical o stationary point if $f'(x)=0$.

A minimum (maximum) is a critical point where $f(x)$ is lower (higher) than at all neighboring points.
There is a third class of critical points: saddle points.

If $f$ is a convex function, this should be the minimum (maximum) of our functions. In other cases it could be a local minimum (maximum) or a saddle point.
There are two problems with numerical derivatives:
+ It is approximate.
+ It is very slow to evaluate (two function evaluations: $f(x + h) , f(x - h)$ ).
Step size
Usually, we multiply the derivative by a step size. This step size (often called alpha) has to be chosen carefully, as a value too small will result in a long computation time, while a value too large will not give you the right result (by overshooting) or even fail to converge.
Analytical derivative
Let's suppose now that we know the analytical derivative. This is only one function evaluation!
Can you write Python code for it?

old_min = 0
temp_min = 15
step_size = 0.01
precision = 0.0001

def f(x):
    return x**2 - 6*x + 5

def f_derivative(x):
    import math
    return 2*x -6

mins = []
cost = []

while abs(temp_min - old_min) > precision:
    old_min = temp_min
    move = f_derivative(old_min) * step_size
    temp_min = old_min - move
    cost.append((3-temp_min)**2)
    mins.append(temp_min)

# rounding the result to 2 digits because of the step size
print "Local minimum occurs at {:3.6f}.".format(round(temp_min,2))


I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
Learning from data
In general, we have:

A dataset ${(\mathbf{x},y)}$ of $n$ examples.
A target function $f_\mathbf{w}$, that we want to minimize, representing the discrepancy between our data and the model we want to fit. The model is represented by a set of parameters $\mathbf{w}$.
The gradient of the target function, $g_f$.

In the most common case $f$ represents the errors from a data representation model $M$.
For example, to fit the model clould be to find the optimal parameters $\mathbf{w}$ that minimize the following expression:
$$ f_\mathbf{w} = \frac{1}{n} \sum_{i} (y_i - M(\mathbf{x}_i,\mathbf{w}))^2 $$
For example, $(\mathbf{x},y)$ can represent:

$\mathbf{x}$: the behavior of a "Candy Crush" player; $y$: monthly payments.
$\mathbf{x}$: sensor data about your car engine; $y$: probability of engine error.
$\mathbf{x}$: finantial data of a bank customer; $y$: customer rating.

If $y$ is a real value, it is called a regression problem.
If $y$ is binary/categorical, it is called a classification problem.

Let's suppose that our model is a one-dimensional linear model $M(\mathbf{x},\mathbf{w}) = w \cdot x $.
Batch gradient descend
We can implement gradient descend in the following way (batch gradient descend):
Can you write Python code for it?

import numpy as np
import random

# f = 2x
x = np.arange(10)
y = np.array([2*i for i in x])

# f_target = 1/n Sum (y - wx)**2
def target_f(x,y,w):
    return np.sum((y - x * w)**2.0) / x.size

# gradient_f = 2/n Sum 2wx**2 - 2xy
def gradient_f(x,y,w):
    return 2 * np.sum(2*w*(x**2) - 2*x*y) / x.size

def step(w,grad,alpha):
    return w - alpha * grad

def BGD_multi_step(target_f,
                   gradient_f,
                   x,
                   y,
                   toler = 1e-6):

    alphas = [100, 10, 1, 0.1, 0.001, 0.00001]
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_ws = [step(w, gradient, alpha) for alpha in alphas]
        next_vals = [target_f(x,y,w) for w in next_ws]
        min_val = min(next_vals)
        next_w = next_ws[next_vals.index(min_val)]
        next_val = target_f(x,y,next_w)
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val

print '{:.6f}'.format(BGD_multi_step(target_f, gradient_f, x, y))

%%timeit
BGD_multi_step(target_f, gradient_f, x, y)

def BGD(target_f, gradient_f, x, y, toler = 1e-6, alpha=0.01):
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_w = step(w, gradient, alpha)
        next_val = target_f(x,y,next_w)
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val

print '{:.6f}'.format(BGD(target_f, gradient_f, x, y))

%%timeit
BGD(target_f, gradient_f, x, y)


I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
Let's start by choosing a random vector and then walking a step in the opposite direction of the gradient vector. We will stop when the difference (in $\mathbf x$) between the new solution and the old solution is less than a tolerance value.
Can you write Python code for it?

# choosing a random vector

import random
import numpy as np

x = [random.randint(-10,10) for i in range(3)]
x

def step(x,grad,alpha):
    return [x_i - alpha * grad_i for x_i, grad_i in zip(x,grad)]

tol = 1e-15
alpha = 0.01
while True:
    grad = gradient_centered(x,f)
    next_x = step(x,grad,alpha)
    if euc_dist(next_x,x) < tol:
        break
    x = next_x
print [round(i,10) for i in x]


I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
Mini-batch Gradient Descent
In code, general batch gradient descent looks something like this:
python
nb_epochs = 100
for i in range(nb_epochs):
    grad = evaluate_gradient(target_f, data, w)
    w = w - learning_rate * grad
For a pre-defined number of epochs, we first compute the gradient vector of the target function for the whole dataset w.r.t. our parameter vector.
Stochastic gradient descent (SGD) in contrast performs a parameter update for each training example and label:
python
nb_epochs = 100
for i in range(nb_epochs):
    np.random.shuffle(data)
    for sample in data:
        grad = evaluate_gradient(target_f, sample, w)
        w = w - learning_rate * grad
Mini-batch gradient descent finally takes the best of both worlds and performs an update for every mini-batch of $n$ training examples:
python
nb_epochs = 100
for i in range(nb_epochs):
  np.random.shuffle(data)
  for batch in get_batches(data, batch_size=50):
    grad = evaluate_gradient(target_f, batch, w)
    w = w - learning_rate * grad
Minibatch SGD has the advantage that it works with a slightly less noisy estimate of the gradient. However, as the minibatch size increases, the number of updates done per computation done decreases (eventually it becomes very inefficient, like batch gradient descent).
There is an optimal trade-off (in terms of computational efficiency) that may vary depending on the data distribution and the particulars of the class of function considered, as well as how computations are implemented.
Can you write Python code for it?

def get_batches(iterable, n = 1):
    current_batch = []
    for item in iterable:
        current_batch.append(item)
        if len(current_batch) == n:
            yield current_batch
            current_batch = []
    if current_batch:
        yield current_batch

%reset
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline

# x: input data
# y: noisy output data

x = np.random.uniform(0,1,2000)

# f = 2x + 0
def f(x): return 2*x + 0

noise_variance =0.1
noise = np.random.randn(x.shape[0])*noise_variance
y = f(x) + noise

plt.plot(x, y, 'o', label='y')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)')
plt.xlabel('$x$', fontsize=15)
plt.ylabel('$t$', fontsize=15)
plt.ylim([0,2])
plt.title('inputs (x) vs targets (y)')
plt.grid()
plt.legend(loc=2)
plt.gcf().set_size_inches((10,3))
plt.show()

# f_target = 1/n Sum (y - wx)**2
def target_f(x,y,w):
    return np.sum((y - x * w)**2.0) / x.size

# gradient_f = 2/n Sum 2wx**2 - 2xy
def gradient_f(x,y,w):
    return 2 * np.sum(2*w*(x**2) - 2*x*y) / x.size

def in_random_order(data):
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

def get_batches(iterable, n = 1):
    current_batch = []
    for item in iterable:
        current_batch.append(item)
        if len(current_batch) == n:
            yield current_batch
            current_batch = []
    if current_batch:
        yield current_batch

def SGD_MB(target_f, gradient_f, x, y, epochs=100, alpha_0=0.01):
    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    epoch = 0
    while epoch < epochs:
        val = target_f(x, y, w)
        if val < min_val:
            min_w, min_val = w, val
            alpha = alpha_0
        else:
            alpha *= 0.9
        np.random.shuffle(data)
   ...


I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
The function we have evaluated, $f({\mathbf x}) = x_1^2+x_2^2+x_3^2$, is $3$ at $(1,1,1)$ and the gradient vector at this point is $(2,2,2)$.
Then, we can follow this steps to maximize (or minimize) the function:

Start from a random $\mathbf{x}$ vector.
Compute the gradient vector.
Walk a small step in the opposite direction of the gradient vector.

It is important to be aware that this gradient computation is very expensive: if $\mathbf{x}$ has dimension $n$, we have to evaluate $f$ at $2*n$ points.

How to use the gradient.
$f(x) = \sum_i x_i^2$, takes its mimimum value when all $x$ are 0.
Let's check it for $n=3$:
Can you write Python code for it?

def euc_dist(v1,v2):
    import numpy as np
    import math
    v = np.array(v1)-np.array(v2)
    return math.sqrt(sum(v_i ** 2 for v_i in v))


I am working on the file "1. PrimitiveTypes_and_operators/.ipynb_checkpoints/objetos simples y operaciones básicas-checkpoint.ipynb".
The first task is:
Ahora cambia el color de tu ventana, del tamaño y color del trazo, y si quieres también crea otra tortuga con tu nombre. Luego intenta dibujar un cuadrado, un triángulo y un hexágono
Ahora vamos a ver qué es lo que estuvimos usando.
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/.ipynb_checkpoints/objetos simples y operaciones básicas-checkpoint.ipynb".
The first task is:
Como podrás darte cuenta, las operaciones y comparaciones que puedes hacer en estos valores son bastantes y puedes ir aumentando la complejidad usando paréntesis y combinando con lo que aprendiste a hacer de operaciones con python
Nota  "==" vs "is"
Cuando quieres saber si un objeto es exactamente ese objeto hay un operador que se llama "is". Nada más que es importante no confundirlo con el operador "==" ya que este a diferencia del "is" te dice si el valor de un objeto equivale a otro. Suena confuso pero tal vez este ejemplo lo clarifique
Can you write Python code for it?

#Ya vimos que El valor de True "equivale" a 1
True == 1

#Ahora nos preguntamos si True "es" 1
True is 1

#y que tal si ahora preguntamos si True "es" True
True is True


I am working on the file "1. PrimitiveTypes_and_operators/.ipynb_checkpoints/objetos simples y operaciones básicas-checkpoint.ipynb".
The first task is:
El planeta Jupyter de Python
Esto que ves aquí es una libreta de Jupyter ejecutando Ipython. Aquí puedes escribir texto (al estilo Markdown) y correr líneas de código que escribas, también hay funciones mágicas que te ayudan a hacer cosas como mostrar las gráficas que creas directamente en la libreta (generalmente Ipython genera una ventana nueva con la imagen lo cual es bastante molesto cuando estás con la libreta...)
Si te gusta leer en inglés y quieres más información puedes consultar el siguiente enlace
Can you write Python code for it?

#Veamos las funciones mágicas de la libreta
%magic


I am working on the file "1. PrimitiveTypes_and_operators/.ipynb_checkpoints/objetos simples y operaciones básicas-checkpoint.ipynb".
The first task is:
Finalmente a Trabajar
Números en Python
Hay dos tipos de número en Python los cuales se llaman "integers" y "floats". Estos números pueden ser utilizados para realizar operaciones matemáticas como lo harías con cualquier calculadora.
Can you write Python code for it?

type(5)

type(5.0)

5+5

5-5.0
#¿Notas algo raro en este resultado?

5.0 * 5

#¿Qué crees que va a dar de resultado esta operación?
5/5

#¿Cómo se puede hacer para que la división nos dé integer en lugar de float?
#Prueba esto
5//5


I am working on the file "1. PrimitiveTypes_and_operators/.ipynb_checkpoints/objetos simples y operaciones básicas-checkpoint.ipynb".
The first task is:
Ipython: la hermana amable de Python
Veamos que tiene que decir Wikipedia acerca de Ipython: ..."IPython es un shell interactivo que añade funcionalidades extra al modo interactivo incluido con Python, como resaltado de líneas y errores mediante colores, una sintaxis adicional para el shell, autocompletado mediante tabulador de variables, módulos y atributos; entre otras funcionalidades"...
Como bien dice la wikipedia, ipython es una variante de python que es mucho más amable con el usuario. Te dice más acerca de los errores que puedas cometer y te ayuda a completar nombres cuando se te olvidan. El código que se corre en esta libreta de Jupyter es con Ipython
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/.ipynb_checkpoints/objetos simples y operaciones básicas-checkpoint.ipynb".
The first task is:
Pregunta 1
¿Qué crees que pasaría si le pedimos a python lo siguiente?
5.0//5.0
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/.ipynb_checkpoints/objetos simples y operaciones básicas-checkpoint.ipynb".
The first task is:
Pregunta 2
¿Crees que obtendrías el mismo resultado si sólo uno de los dividendos fuera un float?
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/.ipynb_checkpoints/objetos simples y operaciones básicas-checkpoint.ipynb".
The first task is:
Pregunta 4
¿Cómo podríamos obtener la potencia de un número y su raíz cuadrada?
Can you write Python code for it?

#Pista: doble estrella

5**2

#Pista: módulo math


I am working on the file "1. PrimitiveTypes_and_operators/.ipynb_checkpoints/objetos simples y operaciones básicas-checkpoint.ipynb".
The first task is:
Strings y todo lo que puedes hacer con palabras
Los strings son cadenas de caracteres, palabras o frases que en python se declaran al ponerlos dentro de comillas, ya sea simples o dobles
Can you write Python code for it?

"Esto es un string"

'Esto también'

#Esto también es un string
'2'

#y en python 3 se pueden concatenar estas series de caracteres sólo poniéndolos continuamente. Hay que notar que un
#espacio también es un caracter...

"Los números"' '"y"' ' 'símbolos'" "'también '"se usan"' ''3456%$#%'


I am working on the file "1. PrimitiveTypes_and_operators/2016-03-02-204917.ipynb".
The first task is:
Como podrás darte cuenta, las operaciones y comparaciones que puedes hacer en estos valores son bastantes y puedes ir aumentando la complejidad usando paréntesis y combinando con lo que aprendiste a hacer de operaciones con python
Nota  "==" vs "is"
Cuando quieres saber si un objeto es exactamente ese objeto hay un operador que se llama "is". Nada más que es importante no confundirlo con el operador "==" ya que este a diferencia del "is" te dice si el valor de un objeto equivale a otro. Suena confuso pero tal vez este ejemplo lo clarifique
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/2016-03-02-204917.ipynb".
The first task is:
El planeta Jupyter de Python
Esto que ves aquí es una libreta de Jupyter ejecutando Ipython. Aquí puedes escribir texto (al estilo Markdown) y correr líneas de código que escribas, también hay funciones mágicas que te ayudan a hacer cosas como mostrar las gráficas que creas directamente en la libreta (generalmente Ipython genera una ventana nueva con la imagen lo cual es bastante molesto cuando estás con la libreta...)
Si te gusta leer en inglés y quieres más información puedes consultar el siguiente enlace
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/2016-03-02-204917.ipynb".
The first task is:
Pregunta 1
¿Qué crees que pasaría si le pedimos a python lo siguiente?
5.0//5.0
Pregunta 2
¿Crees que obtendrías el mismo resultado si sólo uno de los dividendos fuera un float?
Ahora imagina que queremos saber cuál es el residuo de una división. Python tiene un operador que se ve así "%"
Veamos cómo funciona...
Can you write Python code for it?

#¿Cuánto nos queda de residuo si dividimos 25%5?
25%5

#¿Cuánto nos queda de residuo si dividimos 25%7?
25%7


I am working on the file "1. PrimitiveTypes_and_operators/2016-03-02-204917.ipynb".
The first task is:
Pregunta 3
¿Qué pasaría si alguno de los dividendos fuera float?
Pregunta 4
¿Cómo podríamos obtener la potencia de un número y su raíz cuadrada?
Can you write Python code for it?

#Pista: doble estrella

#Pista: módulo math


I am working on the file "1. PrimitiveTypes_and_operators/2016-03-02-204917.ipynb".
The first task is:
Pregunta 5
¿Cómo escribirías tu programa para saludar al mundo?
¡¡Felicidades acabas de escribir y ejecutar tu primer programa!!
Variables: los cajores de python
Hasta ahora hemos visto que podemos hacer operaciones sobre diferentes objetos de python, pero que pasa cuando queremos que una operación o un valor se quede disponible para que los usemos durante todo programa. En python hay algo que se llama variable y está formada por un espacio en el sistema de almacenaje que está asociado a un nombre. Ese espacio contiene una cantidad o información conocida o desconocida, es decir un valor.
Veamos algunos ejemplos.
Can you write Python code for it?

variable_1 = 5 + 5

variable_2 = True

variable_1 + variable_2

cajón = 'Hola a todos'

cajón

print(cajón)

cajón == variable_1


I am working on the file "1. PrimitiveTypes_and_operators/objetos simples y operaciones básicas.ipynb".
The first task is:
Ahora cambia el color de tu ventana, del tamaño y color del trazo, y si quieres también crea otra tortuga con tu nombre. Luego intenta dibujar un cuadrado, un triángulo y un hexágono
Ahora vamos a ver qué es lo que estuvimos usando.
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/objetos simples y operaciones básicas.ipynb".
The first task is:
Finalmente a Trabajar
Números en Python
Hay dos tipos de número en Python los cuales se llaman "integers" y "floats". Estos números pueden ser utilizados para realizar operaciones matemáticas como lo harías con cualquier calculadora.
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/objetos simples y operaciones básicas.ipynb".
The first task is:
Pregunta 2
¿Crees que obtendrías el mismo resultado si sólo uno de los dividendos fuera un float?
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/objetos simples y operaciones básicas.ipynb".
The first task is:
Pregunta 3
¿Qué pasaría si alguno de los dividendos fuera float?
Can you write Python code for it?


I am working on the file "1. PrimitiveTypes_and_operators/objetos simples y operaciones básicas.ipynb".
The first task is:
Pregunta 4
¿Cómo podríamos obtener la potencia de un número y su raíz cuadrada?
Can you write Python code for it?

#Pista: doble estrella

5**2

25**0.5

#Pista: módulo math


I am working on the file "1. PrimitiveTypes_and_operators/objetos simples y operaciones básicas.ipynb".
The first task is:
Puedes crear espacios que luego vas a llenar en estos strings con un método que se llama format
Can you write Python code for it?

'{} of the {}'.format('75%', 'World')

# También puedes usar palabras clave si necesitas poner una misma palabra muchas veces

'{sujeto} camina, {sujeto} juega, {sujeto} duerme con todos {predicado}'.format(sujeto = 'El perro', predicado = 'nosotros')

print('Esta función imprime')


I am working on the file "1. PrimitiveTypes_and_operators/objetos simples y operaciones básicas.ipynb".
The first task is:
Variables: los cajones de python
Hasta ahora hemos visto que podemos hacer operaciones sobre diferentes objetos de python, pero que pasa cuando queremos que una operación o un valor se quede disponible para que los usemos durante todo programa. En python hay algo que se llama variable y está formada por un espacio en el sistema de almacenaje que está asociado a un nombre. Ese espacio contiene una cantidad o información conocida o desconocida, es decir un valor.
Veamos algunos ejemplos.
Can you write Python code for it?


I am working on the file "1. Python (Intermediate)  Exploring Gun Deaths in the US/Basics.ipynb".
The first task is:
Finding
I have founded out, that some racial categories in USA have higher gun-related homicide rate than other races. For example, at least as evidenced by the statics, that people of Black rice commit gun-related homicide 10 times more people of White race or 4 times more people of Hispanic race.
Are the any link between month and homicide rate in USA?
Let figure out that!
Can you write Python code for it?

month_homicide_rate = {}
months = [int(each[2]) for each in data]
for i, each in enumerate(months):
    if intents[i] == 'Homicide':
        if each not in month_homicide_rate:
            month_homicide_rate[each] = 0
        else:
            month_homicide_rate[each] += 1
month_homicide_rate

def months_diff(input_dict):
    max_value = 0
    max_key = 0
    min_value = input_dict[1]
    min_key = 0

    for key, value in input_dict.items():
        if value > max_value:
            max_value = value
            max_key = key
        if value < min_value:
            min_value = value
            min_key = key
    gap = round((max_value / min_value), 2)

    print ('max month is',max_key,'has',max_value,'and min month is',min_key,'has',min_value,'. The gap between min and max months is',gap,'!')

months_diff(month_homicide_rate)


I am working on the file "1. Python (Intermediate)  Exploring Gun Deaths in the US/Basics.ipynb".
The first task is:
The sex and race columns contain potentially interesting information on how gun deaths in the US vary by gender and race. Exploring both of these columns can be done with a similar dictionary counting technique to what we did earlier.
Can you write Python code for it?

sex_counts = {}
race_counts = {}

for each in data:
    sex = each[5]
    if sex in sex_counts:
        sex_counts[sex] += 1
    else:
        sex_counts[sex] = 1

for each in data:
    race = each[7]
    if race in race_counts:
        race_counts[race] += 1
    else:
        race_counts[race] = 1
print(race_counts)
print(sex_counts)


I am working on the file "1. Python Basics.ipynb".
The first task is:
A for loop can also be used to iterate through a dictionary:
Can you write Python code for it?

phoneBook = {"Bob" : "555-555", "Jake" : "555-556", "John" : "555-565", "Jane" : "555-655"}
for name in phoneBook.keys():
    print(name + " : " + phoneBook[name])


I am working on the file "1. Python Basics.ipynb".
The first task is:
A major difference between lists and strings is that lists can be modified while strings cannot. This is referred to as mutability. Lists are mutable; strings are immutable.
String formatting
One particularly useful string method is format. The format method is used to construct strings by inserting values into template strings. Consider this example.
Can you write Python code for it?

name = "Shantanu"
lang = "Python"
ver = 3.6
message = "Hi {}, Welome to coding with {} v{}".format(name, lang, ver)
print(message)


I am working on the file "1. Python Basics.ipynb".
The first task is:
Alternatively, a for loop can be used like a while loop with a definite number of repetitions using the range() function. The range() function returns a list of integers from 0 up till but not including the passed argument.
Can you write Python code for it?

for i in range(5):  # range(5) returns [0, 1, 2, 3, 4]
    print(i)


I am working on the file "1. Python Basics.ipynb".
The first task is:
An example of a sentinel controlled loop:
Can you write Python code for it?

answer = 5
guess = int(input("Guess a number from 1 - 10: "))
while guess != answer:
    guess = int(input("Wrong! Guess a number from 1 - 10: "))
print("Correct!")


I am working on the file "1. Python Basics.ipynb".
The first task is:
Docstrings are surrounded by triple quotes, """. The first line of the docstring is a brief explanation of the function's purpose. If you feel that this is sufficient documentation you can end the docstring at this point, single line docstrings are perfectly acceptable. If you think that the function is complicated enough to warrant a longer description, you can add a more thorough paragraph after the one line summary.
The next element of a docstring is an explanation of the function's arguments. Here you list the arguments, state their purpose, and what types the arguments should be.
Each of these pieces of the docstring is optional, as is the docstring itself. Remember though, it's always easier to write code than to read it! If you can make things easier for your collaborators (which includes future you!) to read your code, then you should.
You can read a more thorough explanation of docstring conventions at https://www.python.org/dev/peps/pep-0257/.

Default Arguments
We can specify default values for arguments of a function. However, these arguments must be placed at the end of the argument list. To understand this useful flexibility that python provides lets look at the famous star box problem and modify it.
Starbox problem:  The function takes two arguments, width and height, that specify how many characters wide the box is and how many lines tall it is.
Can you write Python code for it?

def box(width, height):
    """print a box made up of asterisks.

    width: width of box in characters, must be at least 2
    height: height of box in lines, must be at least 2
    """
    print('*' * width) # print top edge of box

    # print sides of box
    for _ in range(height-2):
        print('*' + " " * (width-2) + '*')

    print('*' * width) # print bottom edge of box

box(5,5)


I am working on the file "1. Python Basics.ipynb".
The first task is:
Functions
Functions allow you to divide your code into chunks that do specific subtasks, which make your code more readable and reusable.
```python
Define the function
def functionName():
    # Function Body
functionName()  # Call the function
```

Parameters & Arguments
Functions can be defined with parameters, which are essentially variables that are set when the function is called using arguments. This allows functions to be reused with different data values.
Can you write Python code for it?

def sum(a, b):  # a & b are parameters
    c = a + b
    print(c)

sum(3, 2)       # 3 & 2 are arguments
sum(32, 65)     # 32 & 65 are arguments


I am working on the file "1. Python Basics.ipynb".
The first task is:
Getting User Input
In Python, we can get input from the user and store this in variables using the input() function. The function prints whatever prompt is passed to it and waits for the user input. Once the Enter key is pressed, it stores the user input into the variable as a string.
Can you write Python code for it?

name = input("Enter your name: ")
print(name)

age = int(input("Enter your age: "))
print(age)


I am working on the file "1. Python Basics.ipynb".
The first task is:
Handling the KeyError
There are two ways to avoid the KeyError problem. Examples below:
Can you write Python code for it?

if "Bob" in ages :
    print("Bob is already in dictionary.")
else :
    print("Bob is not in dictionary.")

if ages.get("Rob") is None:
    print("Rob is not in dictionary.")
else :
    print("Rob is already in dictionary.")


I am working on the file "1. Python Basics.ipynb".
The first task is:
If you want to access a part of a string, Python allows you to slice the string using the start and end indices.
Can you write Python code for it?

s = "hello"
print(s[1:4])   # "ell"
print(s[1:])    # "ello"
print(s[:3])    # "hel"
print(s[:])     # "hello"


I am working on the file "1. Python Basics.ipynb".
The first task is:
It is often useful to have more control over the look of your output. Fortunately, Python provides us with an alternative called formatted strings. A formatted string is a template in which words or spaces that will remain constant are combined with placeholders for variables that will be inserted into the string.
Can you write Python code for it?

print(name, "is", age, "years old.")
print("%s is %d years old." % (name, age))


I am working on the file "1. Python Basics.ipynb".
The first task is:
It is possible to convert a variable from one type to another if the conversion is compatible. For example, converting "123" (string) to 123 (integer) is possible but "hello" cannot be converted to an integer.
Can you write Python code for it?

strA = "123"
intA = int(strA)      # 123
floatA = float(strA)  # 123.0


I am working on the file "1. Python Basics.ipynb".
The first task is:
Lists
Instead of variables holding individual values, it is also possible to create a list of values in Python. Each item in the list is called an element and can be accessed individually using a zero-based index.
( For more information look into Python Advanced notebook. )
Can you write Python code for it?

listOfNums = [1, 2, 3, 4, 5]
print(listOfNums)
print(listOfNums[1])
print(len(listOfNums))


I am working on the file "1. Python Basics.ipynb".
The first task is:
Multiple assignment
It is also possible to assign two variables on a single line:
Can you write Python code for it?

#These two assignments can be abbreviated
savings = 514.86
salary = 320.51

#Using multiple assignment
savings, salary = 514.86, 320.51


I am working on the file "1. Python Basics.ipynb".
The first task is:
Python comes with a lot of useful built-in methods for strings.
Some commons ones are listed below. You can find more at String Methods
|Method Name|Use|Explanation|
|:----------|:--|:----------|
|center |astring.center(w)  |Returns a string centered in a field of size w|
|count  |astring.count(item)|Returns the number of occurrences of item in the string|
|ljust  |astring.ljust(w)   |Returns a string left-justified in a field of size w|
|lower  |astring.lower()    |Returns a string in all lowercase|
|rjust  |astring.rjust(w)   |Returns a string right-justified in a field of size w|
|find   |astring.find(item) |Returns the index of the first occurrence of item|
|split  |astring.split(schar)|Splits a string into substrings at schar|
Can you write Python code for it?

print(s.lower())                   # returns lowercase version of string
print(s.upper())                   # returns uppercase version of string
print(s.find("abc"))               # searches the string for "abc" and returns the first index where it begins or -1
print(s.replace("old", "new"))     # returns a string with all occurrences of "old" replaced with "new"
print(s.center(10))                # returns the string centered in a 10 character space
print(s.split('e'))                # returns the an array with the string characters split on the occurences of e
print(s.islower())
print(s.title())
print(s.count("l"))


I am working on the file "1. Python Basics.ipynb".
The first task is:
Python strings also include a format method that can be used in conjunction with a new Formatter class to implement complex string formatting. More about these features can be found in the Python library reference manual.

Arithmetic Operations
Variables can be used to perform simple arithmetic operations. For example:
Can you write Python code for it?

a = 4
b = 5
c = a + b	# addition
d = a - b	# subtraction
e = a * b	# multiplication
f = a / b	# division
g = a % b	# modulus (remainder)


I am working on the file "1. Python Basics.ipynb".
The first task is:
Reassignment Operators
Because increment and re-assign operation is very common, Python includes special operators for it:
Can you write Python code for it?

manila_pop = 1780148
print(manila_pop)
manila_pop += 1675 # increase the value of manila_pop by 1675
manila_pop -= 250  # decrease the value of manila_pop by 250
manila_pop *= 0.9  # decimate manila_pop
manila_pop /=  2   # approximate the female population of Manila


I am working on the file "1. Python Basics.ipynb".
The first task is:
Statement: Each line of code in a Python program is called a statement. Python interprets statements and runs them one by one.
Comments: The # symbol indicates a comment and anything after # is ignored by the computer. Comments provide information to improve code readability.
Built-in Functions: Python comes with many built-in functions like print() & input() to help you in writing your code.

Variables
What are variables?
Recall variables from algebra — x = 5. In programming, variables are used to store data temporarily to make it easier to refer to them. The value referred to by a variable can be updated as you execute the program using the assignment operator i.e. =.
Can you write Python code for it?

a = 3       # simple assignment
a = a + 1   # RHS is evaluated first
print(a)    # 4


I am working on the file "1. Python Basics.ipynb".
The first task is:
The % operator is a string operator called the format operator. The left side of the expression holds the template or format string, and the right side holds a collection of values that will be substituted into the format string.
Note : the number of values in the collection on the right side corresponds with the number of % characters in the format string. Values are taken—in order, left to right—from the collection and inserted into the format string.
|Character|Output Format|
|:--------|:------------|
|d, i|Integer|
|u|Unsigned integer|
|f|Floating point as m.ddddd|
|e|Floating point as m.ddddde+/-xx|
|E|Floating point as m.dddddE+/-xx|
|g|Use %e for exponents less than −4 or greater than +5, otherwise use %f|
|c|Single character|
|vs|String, or any Python data object that can be converted to a string by using the str function.|
|%|Insert a literal % character|
In addition to the format character, you can also include a format modifier between the % and the format character.
Format modifiers may be used to left-justify or right-justifiy the value with a specified field width.
Modifiers can also be used to specify the field width along with a number of digits after the decimal point.
|Modifier|Example|Description|
|:-------|:------|:----------|
|number|%20d|Put the value in a field width of 20|
|-|%-20d|Put the value in a field 20 characters wide, left-justified|
|+|%+20d|Put the value in a field 20 characters wide, right-justified|
|0|%020d|Put the value in a field 20 characters wide, fill in with leading zeros.|
|.|%20.2f|Put the value in a field 20 characters wide with 2 characters to the right of the decimal point.|
|(name)|%(name)d|Get the value from the supplied dictionary using name as the key.|
Can you write Python code for it?

price = 24
item = "banana"
print("The %s costs %d cents"%(item,price))
print("The %+10s costs %5.2f cents"%(item,price))
print("The %+10s costs %10.2f cents"%(item,price))
print()
itemdict = {"item":"banana","cost":24}
print("The %(item)s costs %(cost)7.1f cents"%itemdict)


I am working on the file "1. Python Basics.ipynb".
The first task is:
The boolean data type, implemented as the Python bool class, will be quite useful for representing truth values. The possible state values for a boolean object are True and False with the standard boolean operators, and, or, and not.
Can you write Python code for it?

print(True)
print(False)
print(False or True)
print(not (False or True))
print(True and True)


I am working on the file "1. Python Basics.ipynb".
The first task is:
There are two main built-in numeric classes that implement the integer and floating point data types.
These Python classes are called int and float.
The standard arithmetic operations : +, -, *, /, and ** (exponentiation), can be used with parentheses forcing the order of operations away from normal operator precedence.Other very useful operations are the remainder modulo operator, %, and integer division, //.
Note : when two integers are divided, the result is a floating point. The integer division operator actually rounds down the number to the closest integer.(Note : Even with Negative number)
Can you write Python code for it?


I am working on the file "1. Python Basics.ipynb".
The first task is:
Tuples
Tuples are very similar to lists in that they are heterogeneous sequences of data. The difference is that a tuple is immutable, like a string. A tuple cannot be changed. Tuples are written as comma-delimited values enclosed in parentheses. As sequences, they can use any operation described above.
Can you write Python code for it?

myTuple = (2,True,4.96)
print(myTuple)

len(myTuple)
print(myTuple[0])
print(myTuple * 3)
print(myTuple[0:2])


I am working on the file "1. Python Basics.ipynb".
The first task is:
for Loop
A for loop can be used to iterate through a list and perform tasks accordingly.
python
for var in someList:
    # Loop Body
In every iteration of the loop, the var variable will be assigned the next value in the list. For example, the code below will print all the elements in the list in separate lines:
Can you write Python code for it?

numList = [2, 1, 4, 5, 3]
for num in numList:
    print(num)


I am working on the file "1. Python Introduction/Exploring Gun Deaths in the US/Exploring Gun Deaths in the US.ipynb".
The first task is:
Death by race
We have learned that black people are much more likely to get killed by gun than people of other races.
For deth rate by race, it could be interesting to evaluate other kinds of death types by race.
Other steps
There are many things to explore in this data set. It is (sadly) interesting set because of recent development in United States.
Can you write Python code for it?

for death_type in set([row[3] for row in data]):
    race_counts = extract_counts([row for row in data if row[3] == death_type],
                                 lambda row: row[7])
    plot_counts(race_counts, "race (%s)" % death_type)

    race_rate_by_100k = {}
    for key in race_counts:
        race_rate_by_100k[key] = float(race_counts[key]) / race_percentage[key] * 100000

    plot_counts(race_rate_by_100k, "race rate by 100 000 (%s)" % death_type)


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
A "prettier" representation of the cited references is available in the citations attribute.
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Additional indexes are located in the indices attribute. The code-cell below shows which fields are already indexed.
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
All of the Papers in the Corpus are stored by wosid in the indexed_papers attribute. The code cell below shows the first ten Papers with their indexing keys.
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Each Paper should also have a date, journal, and wosid (WoS accession ID). Many will also have dois. Note that we can access the attributes of each Paper using . notation:
Can you write Python code for it?

# corpus[500] gets a Paper, and ``.date`` gets the date attribute.
print 'Date:'.ljust(20), corpus[500].date
print 'Journal:'.ljust(20), corpus[500].journal
print 'WoS accession ID:'.ljust(20), corpus[500].wosid
print 'DOI:'.ljust(20), corpus[500].doi


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Each Paper will also have authors. Tethne represents author names as "tuples" of the form (last, first). Depending on the record, first might be first and middle initials, or first and middle names.
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Each cited reference is represented by what we call an 'ayjid': it contains the author name, year of publication, and the journal in which it was published. Every Paper has an 'ayjid'.
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Indexing
The most important functionality of the Corpus is indexing. Indexing provides a way of looking up Papers by specific attributes, e.g. by the year in which they were published, or by author.
Each Corpus has a single "primary" index. For WoS data, the wosid field (WoS accession ID) is used by default, since every WoS record has one. You can see which field was used as the primary index by accessing the .index_by attribute of the Corpus.
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Introduction to Tethne: Working with data from the Web of Science
In this notebook we will take our first steps with the Tethne Python package. We'll parse some bibliographic records from the ISI Web of Science, and take a look at the Corpus class and its various features. We'll then use some of the functions in tethne.networks to generate some simple networks from our bibliographic dataset.
Methods in Digital & Computational Humanities
This notebook is part of a cluster of learning resources developed by the Laubichler Lab and the Digital Innovation Group at Arizona State University as part of an initiative for digital and computational humanities (d+cH). For more information, see our evolving online methods course at https://diging.atlassian.net/wiki/display/DCH.
Getting Help
Development of the Tethne project is led by Erick Peirson. To get help, first check our issue tracking system on GitHub. There, you can search for questions and problems reported by other users, or ask a question of your own. You can also reach Erick via e-mail at erick.peirson@asu.edu.
Using this notebook
This is an interactive Python notebook. Most of the content is just marked-down text, like this paragraph, that provides expository on some aspect of the Tethne package. Some of the cells are "code" cells, which look like this:
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Now use the coauthors function to create the network. We need provide it only our Corpus:
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Simple Networks
One of the core features of Tethne is a set of functions for building networks from bibliographic datasets. These functions are located in the tethne.networks subpackage. In this section, we'll build a coauthorship network.
The first step is to import the networks subpackage.
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Tethne uses a package called NetworkX to build networks. All of the network-building functions return NetworkX Graph objects. We can see how large our network is using the order() and size() methods:
Can you write Python code for it?

print coauthor_graph.order()    # Number of nodes.
print coauthor_graph.size()     # Number of edges.


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
There are several things to notice in the output above. First, each Paper should (generally) have a title:
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
Unlike other bibliographic datasets, WoS data contain the cited references of each Paper. Each cited reference is represented as a Paper:
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
We can "tune" this function by increasing or decreasing min_weight to yield more or less dense graphs. order() (the number of nodes) and size() (the number of edges) give us a sense of the density of the graph.
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
We can create a new index using the index() method. For example, to index Papers by date, we could do:
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
We can look up Papers using the name of an indexed field and some value. For example, to see all of the Papers in which ('MAIENSCHEIN', 'J') is an author, we could do:
Can you write Python code for it?

for paper in corpus[('authors', ('MAIENSCHEIN', 'J'))]:
    print paper.date, paper.title


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
We can use the to_graphml() function once again to write the graph to disk, so that we can visualize it in Cytoscape.
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
We use this function just like the coauthors() function -- passing the Corpus are our first argument -- but we can also pass additional arguments. min_weight indicates that two Papers must share at least three cited references to be coupled. node_attrs tells the function to add additional information to each node; in this case, 'date' and 'title'.
Can you write Python code for it?

coupling_graph = bibliographic_coupling(corpus, min_weight=3, node_attrs=['date', 'title'])


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
You can execute the code in a code cell by clicking on it and pressing Shift-Enter on your keyboard, or by clicking the right-arrow "Run" button in the toolbar at the top of the page. The cell below will automatically be selected, so you can run many cells in quick succession by repeatedly pressing Shift-Enter (or the "Run" button). It's a good idea to run all of the code cells in order, from the top of the tutorial, since many commands later in the tutorial will depend on earlier ones.
Play!!
As we work through the notebook, you'll need to modify certain values depending on where your data is located. You should also experiment! Try changing the parameters in the functions demonstrated below, and re-run the code-cell to see the result. That's what's great about iPython notebooks: you can play around with specific chunks of code without having to re-run the entire script.
Getting Bibliographic Data from the ISI Web of Science
The ISI Web of Science is a proprietary database owned by Thompson Reuters. It is one of the oldest and most comprehensive scientific bibliographic databases in existence. If you are affiliated with an academic institution, you may have access to this database via an institutional license.
For the purpose of this tutorial, you can download a practice dataset from here. Move the downloaded zip to a place where you can find it, and uncompress its contents. You'll need the full path to the uncompressed dataset.
Perform a search for literature of interest using the interface provided.
Your search criteria will be informed by the objectives of your research
project. If you are attempting to characterize the development of a research
field, for example, you should choose terms that pick out that field as uniquely
as possible (consider using the Publication Name search field). You can also
pick out literatures originating from particular institutions, by using the
Organization-Enhanced search field.

Note also that you can restrict your research to one of three indexes in the Web
of Science...
Can you write Python code for it?


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
You can see how many records were loaded from your data file by evaluating the len of the Corpus.
Can you write Python code for it?

print 'Loaded %i records!' % len(corpus)


I am working on the file "1. Working with data from the Web of Science.ipynb".
The first task is:
to_graphml() accepts two arguments: the graph itself, and a string with the path to the output file (that will be created). In the example below, I just put the graph on my desktop.
Can you write Python code for it?


I am working on the file "1. Визуализация.ipynb".
The first task is:

Задание 2: Кто дерется лучше всех?

Правильно сосчитайте количество побед и поражений у каждого из королей (attacker_king, defender_king).
Покажите их статистику как stacked bar plot, отсортировав столбцы по общему количество сражений.
Can you write Python code for it?

#Выберем для наглядности часть таблицы, где только атакующие и защитники
attDef = df[ ['attacker_king', 'defender_king', 'attacker_outcome'] ]

#Отсортируем сначала по attacker_king, внутри attacker_king - по defender_king
attDef = attDef.sort_values(by=['attacker_king', 'defender_king', 'attacker_outcome'], ascending=[False, False, False])

#######################Победы########################################
#Выберем всех победителей-нападавших
attWin = attDef[ ['attacker_king', 'attacker_outcome'] ]
attWin = attWin[attWin.attacker_outcome == 'win']
attWin = attWin[ ['attacker_king'] ]
attWin = attWin.rename(columns={'attacker_king': 'king'}) #переименуем для дальнейшего объединения
attWin = attWin.drop([22,29]) #Удалим NaN

#Выберем всех победителей-защищавшихся
defWin = attDef[ ['defender_king', 'attacker_outcome'] ]
defWin = defWin[defWin.attacker_outcome == 'loss']
defWin = defWin[ ['defender_king'] ]
defWin = defWin.rename(columns={'defender_king': 'king'}) #переименуем для дальнейшего объединения

#Объединим таблицы (фреймы)
framesWin = [attWin, defWin]
resultWin = pd.concat(framesWin)
#######################Поражения######################################
#Выберем всех Проигравших-нападавших
attLoss = attDef[ ['attacker_king', 'attacker_outcome'] ]
attLoss = attLoss[attLoss.attacker_outcome == 'loss']
attLoss = attLoss[ ['attacker_king'] ]
attLoss = attLoss.rename(columns={'attacker_king': 'king'}) #переименуем для дальнейшего объединения

#Выберем всех Проигравших-защищавшихся
defLoss = attDef[ ['defender_king', 'attacker_outcome'] ]
defLoss = defLoss[defLoss.attacker_outcome == 'win']
defLoss = defLoss[ ['defender_king'] ]
defLoss = defLoss.rename(columns={'defender_king': 'king'}) #переименуем для дальнейшего объединения
defLoss = defLoss.drop([22,29, 24]) #Удалим NaN

#Объединим таблицы (фреймы)
framesLoss = [attLoss, defLoss]
resultLoss = pd.concat(framesLoss)

#########################stacked bar plot#########################
#####################################Ну или проще,...


I am working on the file "1. Визуализация.ipynb".
The first task is:

Задание 3: Земля, залитая кровью

Представим, что в битве столкнулись отряды в 5000 и 15000 человек. Будем считать потери равными 2*min(5000, 15000) = 10000. То есть, грубо говоря, всегда побеждает большинство, но из меньшей армии солдат с собой унесет еще хотя бы одного.
Теперь, зная, как высчитывать потери после боя, выясните, в каком конкретно регионе (region) и на какой местности (location) полегло больше людишек. Два графика: один - по регионам, второй - по месностям.
Проиллюстрируйте результаты с помощью pie chart.
Can you write Python code for it?

cleaned = df[df.attacker_size.notnull() & df.defender_size.notnull()].copy()
cleaned['losses'] = 2*np.min(np.array([cleaned.attacker_size, cleaned.defender_size]), axis=0)
f,ax = plt.subplots(1,1, figsize=(15, 15));
cleaned.groupby(['region']).agg(sum).plot.pie(y='losses', ax=ax);

f,ax = plt.subplots(1,1, figsize=(15, 15));
cleaned.groupby(['location']).agg(sum).plot.pie(y='losses', ax=ax);


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
...and the Lancaster stemmer:
Can you write Python code for it?

lancaster = nltk.LancasterStemmer()
print [lancaster.stem(token) for token in tokens][:50]


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
...and the Snowball stemmer (which is actually a collection of stemmers for a whole bunch of different languages):
Can you write Python code for it?

snowball = nltk.SnowballStemmer("english")
print [snowball.stem(token) for token in tokens][:50]


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
A better way is to split on any whitespace, and to separate punctuation from adjacent tokens. The nltk package has a tokenizer called the Penn Treebank Tokenizer that does the job.
Can you write Python code for it?

# Shortcut for Penn Treebank Tokenizer.
from nltk import word_tokenize

tokens = word_tokenize(raw_text)

tokens


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
And here's the top 20 tokens when we apply our processing pipeline:
Can you write Python code for it?

proccessed_text = nltk.Text([normalize_token(token) for token
                             in word_tokenize(raw_text)
                             if filter_token(token)])
proccessed_text.plot(20)


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
Let's compare the effect on our token distributions. Here's the top 20 tokens when we tokenize only:
Can you write Python code for it?

unprocessed_text = nltk.Text(word_tokenize(raw_text))
unprocessed_text.plot(20)


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
Read a file from disk
Can you write Python code for it?

# Try changing this path to point to a different text file.
raw_text = nltk.load('../data/lorem.txt')

raw_text


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
Side-by-side comparisons
Here's a side-by-side comparison of the stemmers shown above.
Can you write Python code for it?

stemmers = [porter.stem, lancaster.stem, snowball.stem, wordnet.lemmatize]
stemmer_labels = ['original', 'porter', 'lancaster', 'snowball', 'wordnet']
print '\t'.join([t.ljust(8) for t in stemmer_labels])
print '--'*40
for token in tokens[:30]:
    print token.ljust(8),
    for stemmer in stemmers:
        print '\t', stemmer(token).ljust(8),
    print

words = ['sustain', 'sustenance', 'sustaining', 'sustains',
         'sustained', 'sustainable', 'sustainability']
stemmers = [porter.stem, lancaster.stem, snowball.stem, wordnet.lemmatize]
stemmer_labels = ['original', 'porter', 'lancaster', 'snowball', 'wordnet']
print '\t'.join([t.ljust(8) for t in stemmer_labels])
print '--'*40
for token in words:
    print token.ljust(8),
    for stemmer in stemmers:
        print '\t', stemmer(token).ljust(8),
    print

wordnet.lemmatize('knows'), wordnet.lemmatize('sleeps'), wordnet.lemmatize('types')


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
Similarly, the dispersion_plot() shows the relative position in which specific tokens occur.
Can you write Python code for it?

text.dispersion_plot(['ipsum', 'orci', 'sem', 'justo', 'arcu', 'Fusce', 'fusce'])


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
Stemming
Stemming is another useful normalization procedure. Sometimes (depending on our research question), distinguishing word forms like "evolutionary" and "evolution" is not desirable. Stemming basically involves chopping off affixes so that only the "root" or "stem" of the word remains.
The NLTK package provides several different stemmers.
Can you write Python code for it?

raw_text = nltk.load('../data/abstract.txt')
tokens = word_tokenize(raw_text.lower())
print tokens[:50]    # Only the first 50 tokens.


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
We can filter out tokens in the stoplist like so:
Can you write Python code for it?

print [token for token in tokens if token not in stoplist][:20]


I am working on the file "1.0 Reading Texts/1.0. Reading texts.ipynb".
The first task is:
We can find all of the occurrences of a token in the document using the concordance() method.
Can you write Python code for it?


I am working on the file "1.0_Control_Flows.ipynb".
The first task is:
break and continue statements

Borrowed from C
In case you don't know... In the loop when encounter:
break: the loop is immediately terminated and the program control resumes at the next statement following the loop.
Can you write Python code for it?

for n in [1, 2, 3, 4, 5]:
    if n % 2 == 0:
        print 'Found an even number', n
        break


I am working on the file "1.0_Control_Flows.ipynb".
The first task is:
exercise
Calculate dot product of two vector [1, 5, 8, 9, 10] and [9, 2, 5, 7, 3]
Can you write Python code for it?

x = [1, 5, 8, 9, 10]
y = [9, 2, 5, 7, 3]
result = 0

for i in range(len(x)):
    result = result + x[i] * y[i]

print result


I am working on the file "1.0_Control_Flows.ipynb".
The first task is:
for statements

the for statement in python differs with the other languages.

No loop over an arithmetic progression of numbers. e.g: cannot write
for(int i = 0; i < 10; i++)

only loop over the items of any sequence
Can you write Python code for it?

for pet in ['cat', 'dog', 'pig']:
    print 'I own a', pet


I am working on the file "1.0_Control_Flows.ipynb".
The first task is:
if you need to iterate over a sequence of numbers, using built-in function range() function
Can you write Python code for it?

print range(10)

print range(5, 10)

print range(0, 10, 3)


I am working on the file "1.0_Control_Flows.ipynb".
The first task is:
the body of the if is indented
indentation is python 's way to grouping statement (code block). C using brackets for grouping.
each line within a code block must be indented by the same about.
you should use only tab to indent or spaces to indent

There can be zero or more elif parts, and the else part is optional
Can you write Python code for it?

x = -5
if x < 0:
    print "X is negative"

x = 5
if x == 0: print 'X is zero'


I am working on the file "1.1 Charts - Timeseries.ipynb".
The first task is:
Exercise: Add the moving annual average to your chart
Tips:
pd.rolling_mean()
Can you write Python code for it?

# Compute moving average

# Create chart with moving average

# Show chart with moving average


I am working on the file "1.1 Charts - Timeseries.ipynb".
The first task is:
Exercise: Style your plot
Ideas:

Add a title
Add axis labels
Change width and height
Deactivate toolbox or customize available tools
Change line color

Charts arguments can be found: http://bokeh.pydata.org/en/latest/docs/user_guide/charts.html#generic-arguments
Can you write Python code for it?

# Style your timeseries chart

# Show new chart


I am working on the file "1.1 Working with a corpus/1.1. Working with a corpus.ipynb".
The first task is:
1.1 Working with a corpus
Let's scale up from the individual text to whole collections. The term "corpus" is commonly used in text analysis.
The concept of a corpus is helpful because it refers not just to a bunch of texts, but also points to the act of purposeful aggregation that brought those texts together. For example, we use the term "corpus" to refer to the collected works of a particular author. For your research, you may aggregate a corpus based on thematic or temporal criteria which are informed by your research questions and theoretical assumptions. When analyzing a corpus, or interpreting the results of such an analysis, it is important to keep in mind its provenance -- not just its contents.
Can you write Python code for it?

text_root = '../../data/EmbryoProjectTexts/files'

try:
    assert os.path.exists(text_root)
except AssertionError:
    print "That directory doesn't exist!"

documents = nltk.corpus.PlaintextCorpusReader(text_root, 'https.+')

documents.words()


I am working on the file "1.1 Working with a corpus/1.1. Working with a corpus.ipynb".
The first task is:
Conditional Frequencies
We can use metadata to add dimensionality to our texts. To examine the distribution of a token over time, we can use a ConditionalFreqDist (conditional frequency distribution). Just like the FreqDist, the ConditionalFreqDist records the outcomes (samples) of an experiment. A ConditionalFreqDist also records a label, or condition, for each outcome.
In the example below, we examine the word usage of different authors. The conditions are the author names, and the samples are tokens. We will limit our analysis to four specific tokens: 'organism', 'ivf', 'pluripotent', 'supreme' (doing this for all tokens would be pretty costly).
Can you write Python code for it?

focal_tokens = ['organism', 'ivf', 'pluripotent', 'supreme']

authorDist = nltk.ConditionalFreqDist([
        (str(author[0]), normalize_token(token))    # (condition, sample)
         for fileid in corpus.fileids()
         for token in corpus.words(fileids=[fileid])
         for author in metadata[fileid].authors
         if filter_token(token)
            and normalize_token(token) in focal_tokens
    ])

authorDist.tabulate()


I am working on the file "1.1 Working with a corpus/1.1. Working with a corpus.ipynb".
The first task is:
In the figure above, we can see that the top ~40 words occur in around 630 texts. We can see precise values using the most_common() function:
Can you write Python code for it?

document_counts.most_common(10)    # Get the 10 most common words.


I am working on the file "1.1 Working with a corpus/1.1. Working with a corpus.ipynb".
The first task is:
It can be useful to examine the number of texts in which a word occurs, to get a better picture of its distribution over the corpus. We can use a FreqDist for this, too.
We modify our logic slightly: for each document, we conver the list of normalized/filtered tokens into a set. This means that each word will be counted only once per text, even if several tokens are present.
It turns out that there are 628 texts in this corpus...
So these are words that occur in every single text in the corpus.
Can you write Python code for it?


I am working on the file "1.1 Working with a corpus/1.1. Working with a corpus.ipynb".
The first task is:
Normalization and Filtering
We'll use the normalization and filtering procedures from our last notebook. The code-cell below sets up the functions that we defined earlier.
Can you write Python code for it?

wordnet = nltk.WordNetLemmatizer()
from nltk.corpus import stopwords
stoplist = stopwords.words('english')

def normalize_token(token):
    """
    Convert token to lowercase, and stem using the Porter algorithm.

    Parameters
    ----------
    token : str

    Returns
    -------
    token : str
    """
    return wordnet.lemmatize(token.lower())

def filter_token(token):
    """
    Evaluate whether or not to retain ``token``.

    Parameters
    ----------
    token : str

    Returns
    -------
    keep : bool
    """
    token = token.lower()
    return token not in stoplist and token.isalpha() and len(token) > 2


I am working on the file "1.1 Working with a corpus/1.1.2. Extracting plain text from text-embedded PDFs.ipynb".
The first task is:
A better way to open a file is to use a with statement. The basic idea is that we can open the file, do some procedures (in the indented block), and then the file will be automatically closed after the end of the indented block. So:
Can you write Python code for it?

with codecs.open('../data/example.pdf') as f:
    pass    # do something


I am working on the file "1.1 Working with a corpus/1.1.2. Extracting plain text from text-embedded PDFs.ipynb".
The first task is:
But you have to remember to close the file!! If you don't, Bad Things Will Happen.
Can you write Python code for it?


I am working on the file "1.1 Working with a corpus/1.1.2. Extracting plain text from text-embedded PDFs.ipynb".
The first task is:
Extracting plain text from text-embedded PDFs
In Module 1, we briefly touched on the problem of extracting usable plain-text content from PDF documents. In this notebook, we will use a Python package called slate to extract text from PDF documents that have embedded text (see Module 1 for details on what that means).
Start with one file
Like most things in life, you should try to do the whole job all the way through until you're fairly certain of the procedure. So, we'll start with a single file.
First we have to open the file. The open() function creates a file object. You can open a file like this:
Can you write Python code for it?


I am working on the file "1.1 Working with a corpus/1.1.2. Extracting plain text from text-embedded PDFs.ipynb".
The first task is:
Hey, it's text! Notice that the text is actually represented by a list of strings. Each page is represented as a separate string; how nice! If I wanted to keep these pages separate, I could save each one as a separate text file. Or I could stitch them together into a single string, and save the document as one text file.
To stitch the pages together, I could use the the join() method.
Can you write Python code for it?

# This concatenates the pages, with two newlines intervening between them.
joined = "\n\n".join(extracted_text)
joined


I am working on the file "1.1 Working with a corpus/1.1.2. Extracting plain text from text-embedded PDFs.ipynb".
The first task is:
Now instead of \xe2\x80\x99, we see \u2019. Success! If we had gotten the encoding wrong, those characters would not have been transformed successfully. For example:
Can you write Python code for it?