I am working on the file "3d_ml/neural_rendering/Test.ipynb".
The first task is:
Voxelize Mesh
Can you write Python code for it?

voxel_resolution = 50
mesh = trimesh.load(data_dir / 'suzanne.obj')

mesh_voxels = mesh_to_voxels(mesh, voxel_resolution, pad=False)
mesh_voxels.shape

voxels = pred_voxel.reshape(tuple([size]*3))

# show mesh
vertices, faces, normals, _ = skimage.measure.marching_cubes_lewiner(voxels, level=0)
mesh = trimesh.Trimesh(vertices=vertices, faces=faces, vertex_normals=normals)
mesh.show()

# save voxels to file
np.save(data_dir / 'suzanne.npy', np.array(voxels, dtype=np.float16))


I am working on the file "3dgfx.ipynb".
The first task is:
And now we call the triangle_example1 function to plot it:
Can you write Python code for it?


I am working on the file "3dgfx.ipynb".
The first task is:
We have all the components to express $P$ in terms of the points of the triangle ${P_1}{P_2}{P_3}$.
We start at $P_1$ and go some amount $t_2$ in the direction of the vector $P_2 - P_1$. We'll end up at $P1 + t_2(P2 - P1)$. From there we go some amount $t_3$ in the direction of the vector $P_3 - P_1$ and with the proper amounts (or ratios) of $t_2$ and $t_3$ we should be able to end up at $P$.
To visualize it we'll use a slightly different plot. The dashed line goes through point $P$ and runs in the direction of $P_3 - P_1$. We start at $P_1$ and go in the direction of the $P_2 - P_1$ vector until we end up at the dashed line. From there we just have to follow said dashed line until we end up at $P$.
Can you write Python code for it?

plt.figure(figsize=(6,6))
triangle_example1()

f = lambda x: 1/4 * x + 1
x = np.linspace(-1, 5, 5)

# FIXME: Let's just do it engineering style for now
f_x = f(x) - f(P[0]) + P[1]

plt.plot(x, f_x, 'r--')

ax = plt.axes()
ax.annotate(r'$t_2$', (0.25, 2))
ax.annotate(r'$t_3$', (1.25, 2.45))


I am working on the file "4 - Earth Engine and TensorFlow.ipynb".
The first task is:
Build the TensorFlow Model
We start with a helper function to build a simple TensorFlow neural network layer.
Can you write Python code for it?

def make_nn_layer(input, output_size):
  input_size = input.get_shape().as_list()[1]
  weights = tf.Variable(tf.truncated_normal(
      [input_size, output_size],
      stddev=1.0 / math.sqrt(float(input_size))))
  biases = tf.Variable(tf.zeros([output_size]))
  return tf.matmul(input, weights) + biases


I am working on the file "4 - Earth Engine and TensorFlow.ipynb".
The first task is:
Display the local data. This time, for variety, we display it as an NRG false-color image. We can use pyplot to display local numpy arrays.
Can you write Python code for it?

pyplot.imshow(numpy.clip(data[:,:,[3,2,1]] * 3, 0, 1))
pyplot.show()


I am working on the file "4 - Earth Engine and TensorFlow.ipynb".
The first task is:
Initialize the Earth Engine client. This assumes that you have already configured Earth Engine credentials in this Datalab instance. If not, see the "Earth Engine Datalab Initialization.ipynb" notebook.
Can you write Python code for it?


I am working on the file "4 - Earth Engine and TensorFlow.ipynb".
The first task is:
Inspect the Input Data
Load a Landsat image with corresponding Fmask label data.
Can you write Python code for it?


I am working on the file "4 - Earth Engine and TensorFlow.ipynb".
The first task is:
Inspect the Results
Here we dislay the results. The red band corresponds to the TensorFlow output and the blue band corresponds to the labeled training data, so pixels that are red and blue correspond to disagreements between the model and the training data. (There aren't many: look carefully around the fringes of the clouds.)
Can you write Python code for it?

output_image = output_data.reshape([data.shape[0], data.shape[1]])
red = numpy.where(data[:,:,8], output_image, 0.5)
blue = numpy.where(data[:,:,8], data[:,:,7], 0.5)
green = numpy.minimum(red, blue)

comparison_image = numpy.dstack((red, green, blue))
pyplot.figure(figsize = (12,12))
pyplot.imshow(comparison_image)
pyplot.show()


I am working on the file "4 - Earth Engine and TensorFlow.ipynb".
The first task is:
Let's define a helper function to make it easier to print thumbnails of Earth Engine images. (We'll be adding a library with utility functions like this one to the Earth Engine Python SDK, but for now we can do it by hand.)
Can you write Python code for it?

def print_image(image):
  display.display(display.Image(ee.data.getThumbnail({
      'image': image.serialize(),
      'dimensions': '360',
  })))


I am working on the file "4 - Earth Engine and TensorFlow.ipynb".
The first task is:
Now we can use our helper function to quickly visualize the image and label data. The Fmask values are:
0 | 1 | 2 | 3 | 4
:---:|:---:|:---:|:---:|:---:
Clear | Water | Shadow | Snow | Cloud
Can you write Python code for it?

print_image(input_image.visualize(
    bands=['B3', 'B2', 'B1'],
    min=0,
    max=0.3,
))
print_image(input_image.visualize(
    bands=['fmask'],
    min=0,
    max=4,
    palette=['808080', '0000C0', '404040', '00FFFF', 'FFFFFF'],
))


I am working on the file "4 - Earth Engine and TensorFlow.ipynb".
The first task is:
Train the Neural Network
Now train the neural network, using batches of training data drawn randomly from the training data pool. We periodically compute the accuracy against the validation data. When we're done training, we apply the model to the complete input data set.
This simple notebook performs all TensorFlow operations locally. However, for larger analyses you could bring up a cluster of TensorFlow workers to parallelize the computation, all controlled from within Datalab.
Can you write Python code for it?

BATCH_SIZE = 1000
NUM_BATCHES = 1000

with tf.Session() as sess:
  sess.run(tf.initialize_all_variables())

  validation_dict = {
    input: validation_data[:,0:7],
    labels: validation_data[:,7],
  }

  for i in range(NUM_BATCHES):
    batch = training_data[numpy.random.choice(training_size, BATCH_SIZE, False),:]
    train_step.run({input: batch[:,0:7], labels: batch[:,7]})

    if i % 100 == 0 or i == NUM_BATCHES - 1:
      print('Accuracy %.2f%% at step %d' % (accuracy.eval(validation_dict) * 100, i))

  output_data = outputs.eval({input: data_vector[:,0:7]})


I am working on the file "4 - Earth Engine and TensorFlow.ipynb".
The first task is:
We can zoom in on a particular region over on the right side of the image to see some of the disagreements. Red pixels represent comission errors and blue pixels represent omission errors relative to the labeled input data.
Can you write Python code for it?

pyplot.figure(figsize = (12,12))
pyplot.imshow(comparison_image[300:500,600:,:], interpolation='nearest')
pyplot.show()


I am working on the file "4 - Sorting.ipynb".
The first task is:
A little function that takes a string, and returns its last letter.
This will be the key function (takes in 1 value, returns 1 value).
Can you write Python code for it?


I am working on the file "4 - Sorting.ipynb".
The first task is:
As another example, specifying "str.lower" as the key function is a way to force the sorting to treat uppercase and lowercase the same:
Can you write Python code for it?

strs = ['aa', 'BB', 'zz', 'CC']
print sorted(strs, key=str.lower)


I am working on the file "4 - Sorting.ipynb".
The first task is:
Custom Sorting With key
For more complex custom sorting, sorted() takes an optional "key=" specifying a "key" function that transforms each element before comparison. The key function takes in 1 value and returns 1 value, and the returned "proxy" value is used for the comparisons within the sort.
For example with a list of strings, specifying key=len (the built in len() function) sorts the strings by length, from shortest to longest. The sort calls len() for each string to get the list of proxy length values, and the sorts with those proxy values.
Can you write Python code for it?

strs = ['ccc', 'aaaa', 'd', 'bb']
print sorted(strs, key=len)


I am working on the file "4 - Sorting.ipynb".
The first task is:
List Comprehensions
A list comprehension is a compact way to write an expression that expands to a whole list. Suppose we have a list nums [1, 2, 3], here is the list comprehension to compute a list of their squares [1, 4, 9]:
Can you write Python code for it?

nums = [1, 2, 3, 4]
squares = [ n * n for n in nums ]
squares


I am working on the file "4 - Sorting.ipynb".
The first task is:
Now pass key=MyFn to sorted() to sort by the last letter.
Can you write Python code for it?


I am working on the file "4 - Sorting.ipynb".
The first task is:
The above is a very common misunderstanding with sort() -- it does not return the sorted list. The sort() method must be called on a list; it does not work on any enumerable collection (but the sorted() function above works on anything). The sort() method predates the sorted() function, so you will likely see it in older code. The sort() method does not need to create a new list, so it can be a little faster in the case that the elements to sort are already in a list.
Tuples
A tuple is a fixed size grouping of elements, such as an (x, y) co-ordinate. Tuples are like lists, except they are immutable and do not change size (tuples are not strictly immutable since one of the contained elements could be mutable). Tuples play a sort of "struct" role in Python -- a convenient way to pass around a little logical, fixed size bundle of values. A function that needs to return multiple values can just return a tuple of the values. For example, if I wanted to have a list of 3-d coordinates, the natural python representation would be a list of tuples, where each tuple is size 3 holding one (x, y, z) group.
To create a tuple, just list the values within parenthesis separated by commas. The "empty" tuple is just an empty pair of parenthesis. Accessing the elements in a tuple is just like a list -- len(), [ ], for, in, etc. all work the same.
Can you write Python code for it?

tuple = (1, 2, 'hi')
print len(tuple)
print tuple[2]


I am working on the file "4 - Sorting.ipynb".
The first task is:
Tuples are immutable, i.e. they cannot be changed.
Can you write Python code for it?


I am working on the file "4 - Sorting.ipynb".
The first task is:
You can add an if test to the right of the for-loop to narrow the result. The if test is evaluated for each element, including only the elements where the test is true.
Can you write Python code for it?

## Select values <= 2
nums = [2, 8, 1, 6]
small = [ n for n in nums if n <= 2 ]
small

## Select fruits containing 'a', change to upper case
fruits = ['apple', 'cherry', 'bannana', 'lemon']
afruits = [ s.upper() for s in fruits if 'a' in s ]
afruits


I am working on the file "4 - Sorting.ipynb".
The first task is:
You can also pass in your own MyFn as the key function. Say we have a list of strings we want to sort by the last letter of the string.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-0 General pandas Concepts.ipynb".
The first task is:
But now that we have columns as well as an index, we can rename the columns to better describe and query the data.
Can you write Python code for it?

letters.columns = ['lowercase','uppercase','number']

letters.lowercase

letters['lowercase']


I am working on the file "4 - pandas Basics/4-0 General pandas Concepts.ipynb".
The first task is:
Just like a numpy array we can transpose it.
Can you write Python code for it?

letters = letters.transpose()
letters.head()

letters.columns

letters.index


I am working on the file "4 - pandas Basics/4-0 General pandas Concepts.ipynb".
The first task is:
Let’s go ahead and create one. We’ve make it from the lowercase, uppercase, and a number range.
Can you write Python code for it?

letters = pd.DataFrame([lcase, ucase, list(range(26))])
letters


I am working on the file "4 - pandas Basics/4-0 General pandas Concepts.ipynb".
The first task is:
Now don’t worry about the functions that I’m using. We’re going to go over those in detail - I just wanted to introduce the concept.
We’ve got the DataFrame which is like a matrix or series of series’. It also has an index (or multiple indexes).
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-0 General pandas Concepts.ipynb".
The first task is:
Now that we’ve created it. We can see it has an index, that we just talked about, as well as values. When we print these out, they should look similar - just like numpy arrays. Now here is where the series gets powerful.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-0 General pandas Concepts.ipynb".
The first task is:
Now we can query just like we would if an array. You can think of the Series like an extremely powerful array.
We can query either sections or specific values.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-0 General pandas Concepts.ipynb".
The first task is:
We can even set up a date range to associate each letter with a date. Now obviously this isn’t too helpful for the alphabet, but this allows you to do some amazing things once you are analyzing real data.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-0 General pandas Concepts.ipynb".
The first task is:
We’ve got the Series. The Series is like a 1 dimensional array in numpy. It has some helper functions and an index that allows for querying of the data in simple ways.
We can make a simple Series from a numpy array.
Can you write Python code for it?

pd.Series

series_ex = pd.Series(np.arange(26))
series_ex


I am working on the file "4 - pandas Basics/4-0 General pandas Concepts.ipynb".
The first task is:
we can replace the index with our own index. In this example I’ll use the lower case values of ascii characters.
Can you write Python code for it?

import string
lcase = string.ascii_lowercase
ucase = string.ascii_uppercase
print(lcase, ucase)

lcase = list(lcase)
ucase = list(ucase)
print(lcase)
print(ucase)

series_ex.index = lcase

series_ex.index

series_ex


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
Adding them together means each one is added to each one.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
Fillna just fills the blanks with whatever value you specify.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
However things get more complicated when they have different indices. Now when we try and add them it only does so on the overlapping index labels. Often times this may be what we want when we’re analyzing data but other times it’s not. In order to handle that we’ve got to do some reindexing and use fill values.
Can you write Python code for it?

s2.index = list(range(3,8))
s2

s1 + s2

s1.reindex(range(10),fill_value=0) + s2.reindex(range(10),fill_value=0)

s2.index = range(5)

s1 = pd.Series(range(1,4), index= ['a','a','c'])
s1

s2 = pd.Series(range(1,4), index=['a','a','b'])
s2


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
Maps are going to feel familiar from our raw python section, except we can do something a bit more special with the pandas Series Version. We can map it to a dictionary as well. This will perform a look up in the dictionary and return whatever is there.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
Now NaN values are treated differently in numpy than in pandas. In numpy, as we saw earlier if you’ve got an array with a NaN value, things like summary statistics are calculated as NaN.
Can you write Python code for it?

np_array = np.array([1,2,3,np.nan])
np_array

np_array.mean()

pd_series = pd.Series([1,2,3,np.nan])
pd_series


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
Now at times you’re going to want to reindex a Series. What does this mean? Basically that you want to destroy the index you have currently and reset it. Let’s walk through a practical example.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
Pandas Series treat them differently, it just ignores that empty value. We’ll cover filling in those empty values at a later time.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
Sometimes you're going to have to make some new indexes. For example we've got two Series.
Can you write Python code for it?

s1 = pd.Series(np.random.randn(5))
s1

s2 = pd.Series(np.random.randn(5))
s2


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
There are a couple more methods I want to touch on, most specifically map.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
When we concatenate them, we can see we’ve got repeated index values. We can query just like we would normally by these index values, but in all likelihood we’ll want to replace them with a new one.
Can you write Python code for it?

combo[0]

combo.index = range(combo.count())
combo


I am working on the file "4 - pandas Basics/4-3 pandas Series NaNs, Reindexing, filling, mutating and copies, basic mapping.ipynb".
The first task is:
and bfill or backward fill
Can you write Python code for it?

new_combo.bfill()

new_combo[21] = 5

new_combo

new_combo.bfill()

new_combo


I am working on the file "4 - pandas Basics/4-4 pandas DataFrame Basics.ipynb".
The first task is:
Now just like Series, DataFrames have data types, we can get those by accessing the dtypes of the DataFrame which will give us details on the data types we've got.
Can you write Python code for it?

letters.dtypes

letters.index = lcase
letters


I am working on the file "4 - pandas Basics/4-4 pandas DataFrame Basics.ipynb".
The first task is:
Now you’ll see that if these lengths are not the same, we’ll get a ValueError so it’s worth checking to make sure your data is clean before importing or using it to create a DataFrame
Can you write Python code for it?

pd.DataFrame({'lowercase':lcase + [0], 'uppercase':upcase})

letters.head()


I am working on the file "4 - pandas Basics/4-4 pandas DataFrame Basics.ipynb".
The first task is:
This should be familiar because it’s the same way that we transpose ndarrays in numpy.
Of course we can also specify them as explicit columns but passing in a dictionary where the keys are the column names and the values are the lists of each item (or the rows).
Can you write Python code for it?

letters = pd.DataFrame({'lowercase':lcase, 'uppercase':upcase})
letters.head()


I am working on the file "4 - pandas Basics/4-4 pandas DataFrame Basics.ipynb".
The first task is:
We can also just query the index as well. We went over a lot of that in the Series Section and a lot of the same applies here.
We can query by index location or by letters
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-4 pandas DataFrame Basics.ipynb".
The first task is:
We can rename the columns easily and even add a new one through a relatively simple dictionary like assignment. I'll go over some more complex methods later on.
Can you write Python code for it?

letters.columns = ['LowerCase','UpperCase']

np.random.seed(25)
letters['Number'] = np.random.random_integers(1,50,26)

letters


I am working on the file "4 - pandas Basics/4-4 pandas DataFrame Basics.ipynb".
The first task is:
We've seen how to query for one column and multiple columns isn't too much more difficult.
We can get upper and lower case columns
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-4 pandas DataFrame Basics.ipynb".
The first task is:
We’ll be covering a lot of different aspects here but as always we’re going to start with the simple stuff. A simplification of a data frame is like an excel table or sql table. You’ve got columns and rows.
In more specific pandas terms, it's a more powerful list of series. Each column is a Series of data and it just so happens these can have relationships.
You can see that if we just pass in a list of lists it treats them like columns. Of course if that’s an issue we can just transpose it and get we’ll get them as columns.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Alright, now remember we’re just going to use the fillna method to fill it in. Since we’re going to assume I ran 0 miles on those days. We just fill it with 0 and do it inplace. Now our plot looks much better.
Can you write Python code for it?

df.Miles.fillna(0,inplace=True)

df.head()

df.plot()


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Boom now we’ve got the correct dates. Now pandas is pretty good at parsing dates but occasionally it will mess them up. You may have to write a custom date parser but doing so is outside the scope of this course.
Let’s continue with our analysis.
Now that we’ve gotten our dates parsed, let’s try set the Dates as the index. You’ll see I’m using the inplace=True command. This is because typically pandas will try to return a new data frame rather than editing our current one. In place makes so that it edits the current one.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Let’s go ahead and plot it.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Now by default pandas uses matplotlib to plot its graphs. Our first line in this notebook is a magic function that makes these matplotlib graphs be created in our notebook.
Now let's a print a complete table of the data
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Now for a little background on this dataset. This is my exercise data from august 1st 2014 to January 1st 2015. During these several months I recorded the distance every time that I ran and recorded the time the majority of the time that took as well.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Now that we’ve described it, and when we look at the data itself.
I’m going to pause for a second to let you think about this, but basically there’s something that’s a bit irregular about this data. Go ahead and pause, try and explore it a bit and when you’re ready un pause the video.
Alright, think you’ve gotten it?
Well the problem is that our data is recorded on inconsistent days. You’ll see dates that have multiple entries and others that don’t have any entry at all. For instance November.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Now that we’ve printed our the data in our data frame both as a graph and as a table. Let’s try summarizing it a bit better. We can do that better with the describe command. This is going to give us some basic summary statistics about our data, the number of points we have, the mean and standard deviation as well as min and max and quartiles.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Now we can see our 3 columns here, date, miles and time. We can see that we’ve got some NaN values when I didn’t record the time. We saw those in the head command above.
We’ve also go our index which is just numbered.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Now we’re going to dive into our first data set. I’m extremely excited because this is where things actually get fun! We’re going to encounter some real problems that you’ll likely experience all the time during your analysis and work.
So lets get started!
Let’s go ahead and open up the running_data.csv
We can do that with the read csv command provided by pandas. That’s going to read a data frame for us. That we can see right here.
You'll notice there are a lot of other formats that we can use. I'm not going to cover them but they are there.
Can you write Python code for it?

df = pd.read_csv('../data/raw_running_data.csv')
print(type(df))
df.head(10)

?pd.read_csv()


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
Of course there are multiple ways to perform this operation but I’m going to put it in as a join to illustrate and import concepts. Joins. A join is like a SQL join or a venn diagram.
Can you write Python code for it?

from IPython.display import Image
Image(url='http://i.stack.imgur.com/GbJ7N.png')

df.join(d).head(10)

d.join(df).head(10)

df.join(d, how='outer')

df = df.join(d, how='outer')


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
So we performed an outer join. There are other kinds of joins like inner joins, left and right that you can look up in the documentation.
but now we’ve got a new problem, we’ve got a bunch of NaN values in our Miles column, luckily those are easy to fix. Go ahead and pause and try to fix this in your own.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
We can also avoid importing datetime by just using the built in datetime function which creates one in the exact same way.
Now you might be wonder
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
You'll also notice that there are a TON of extra parameters that can be passed into this function, we can skip rows, specify dtypes, if there's compression or not, the index column, how many chunks or rows to read. This function could be a whole video itself. The important part is, when you're reading files, you've got a lot of options. You can do a lot of this processing after the fact and if you have any specific questions please don't hesistate to ask.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-5 pandas DataFrame Intros to File IO, Plotting, Stats and Joins.ipynb".
The first task is:
You’ll see that it prints what it can, it doesn’t understand what our time data is yet so it doesn’t plot it. We’ll address that a bit later.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
How do you think we might go from october to January 1 2015?
Go ahead and give it a try and see if you can figure it out.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
Let’s plot Miles and Minutes together in a scatter plot. Wow that’s linear. Let’s see how correlated they are. We do this with the cor method. We can see that Miles to time are very tightly correlated (using pearson standard correlation coefficients) there are two other correlation methods that you can use, kendall Tau, and Spearman rank correlation.
Can you write Python code for it?

df.corr()

df.corr(method='kendall')

df.corr(method='spearman')


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
Let’s see a histogram of my speeds.
Histograms are a great way of representing frequency data or how much certain things are occuring.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
Now let’s add minutes per mile, we can just divide our two series to get those numbers.
Can you write Python code for it?

df['Minutes'] / df['Miles']

df['Min_per_mile'] = df['Minutes'] / df['Miles']

df.describe()


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
Now let’s do a describe and plot it again.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
Now that we have seconds it would be useful to see minutes too. I find it hard to think in seconds.
So let’s add a new column.
Can you write Python code for it?

df['Minutes'] = df['Seconds'].map(lambda x: x / 60)

df.describe()


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
Now there’s a LOT more you can do with date time indexing but this is about all that I wanted to cover in this video. We will get into more specifics later. By now you should be getting a lot more familiar with pandas and what the ipython + pandas workflow is.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
Now we can plot it again once we set the index.
Can you write Python code for it?

df.set_index('Date', inplace=True)

df.plot()


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
So here’s a standard plot of our Miles again, just a line over time. To add another line to the same plot we just add more details to the box. As I was touching on the rolling values. Let’s talk about the rolling average. Now to do that I pass it a series or a data frame.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
This is a great example of what  np.NaN is so great in pandas. When we do this describe we get answers unlike in numpy and it just disregards the empty values. If we were to fill it in with zeros that would drag down our statistics.
We can see that here. See how much it changes the data when it is filled with 0 which would be incorrect analysis.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
This is because when you save a data frame to a csv it doesn’t label the index column. So now our column is actually the ‘zero’ column. When you use parse_dates you can use either the column name (if available) or the 0-based column index number. This happened because the index didn’t have a name when we saved the last csv.
Can you write Python code for it?

df['Unnamed: 0'].head()

df = pd.read_csv('../data/date_fixed_running_data.csv', parse_dates=['Date'])


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
We can see that along more shorter distances, my speed can vary a lot.
Can you write Python code for it?

df.plot(x='Miles', y='Min_per_mile', kind='scatter')
plt.ylabel("Minutes / Mile")


I am working on the file "4 - pandas Basics/4-6 pandas DataFrame Renaming Cols, Handling NaN Values, Maps, Intermediate Plotting, + Rolling Values, + Basic Date Indexing.ipynb".
The first task is:
seems pretty center in that 7 minutes to 7.5 minute range. Let’s see if we can get more information with more bins which we specify with the bin argument.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
Another interesting question we could ask is what days of the week do I commonly go for runs. Am I faster on certain days or does my speed improve over time relative to the distance that I'm running.
So let's get our days of the week
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
It might be interesting to see the total sum of the amount of runs to try and see any outliers simply because Thursday, Friday, Saturday are close in distances, relatively, but not so much in speed.
We also get access to a lot of summary statistics from here that we can get from the groups.
Can you write Python code for it?

dows.describe()

df.groupby('Day_of_week').mean()

df.groupby('Day_of_week').std()


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
Now let's about getting some summary statistics. I would encourage you to try these on your own. We've learned pretty much everything we need to in order to be able to do these on without guidance and as always if you need clarification just ask on the side.
What was the longest run in miles and minutes that I ran?
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
Now we have our groups and we can start doing groupwise analysis, now what does that mean?
It means we can start answering questions like what is the average speed per weekday or what is the total miles run per weekday?
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
Now we've made our dataset a bit more manageable. We've kind of just got the basics of what we need to perform some groupwise analysis.
Now at this point we're going to do some groupings. This is an extremely powerful part of pandas and one that you'll use all the time.
pandas follows the the Split-Apply-Combine style of data analysis.
Many data analysis problems involve the application of a split-apply-combine strategy, where you break up a big problem into manageable pieces, operate on each piece independently and then put all the pieces back together.
Hadley Wickhan from Rice University:
http://www.jstatsoft.org/v40/i01/paper
Since we're going to want to check things in groups. What I'm going to do is try to analyze each day of the week to see if there are any differences in the types of running that I do on those days.
we'll start by grouping or data set on those weekdays. Basically creating a dictionary of the data where the key is the weekday and the value is the dataframe of all those values.
First let's do this the hard way....
Can you write Python code for it?

for dow in df.Day_of_week.unique():
    print(dow)
    print(df[df.Day_of_week == dow])
    break


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
Plot a graph of the cumulative running distance in this dataset.
Can you write Python code for it?

df.Miles.cumsum().plot()
plt.xlabel("Day Number")
plt.ylabel("Distance")


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
Plot a graph of the cumulative running hours in this data set.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
This is clearly an ugly way to do this and pandas provides a much more simple way of approaching this problem. by creating a groupby object.
But first I'm going to filter out our zero values because they'll throw off our analysis.
Can you write Python code for it?

df['Miles'] = df.Miles[df.Miles > 0]

dows = df.groupby('Day_of_week')
print(dows)


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
We can use an aggregation command to perform an operation to get all the counts for each data frame.
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
We forgot to ignore our null values, so how would we do it by ignoring those?
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
We will do that by creating groups of data frames
We can see that in this sample I run a lot more on the Friday Saturday and Monday. Some interesting patterns. Why don't we try looking at the means and that sort of thing.
But before we get there, at this point, our data frame is getting pretty messy and I think it's worth explaining how to remove columns and add remove rows and columns.
First let's remove the Time column - seeing as we already have minutes and  seconds
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
We will do that by mapping our date column to a the time format we need
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
What about the shortest in miles and minutes that I ran?
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
del will delete it in place
Can you write Python code for it?


I am working on the file "4 - pandas Basics/4-7 pandas DataFrame Summary Statistics, Filtering, Dropping and adding Rows and Columns, Grouping Basics.ipynb".
The first task is:
you can get specific groups by using the get_group method.
Can you write Python code for it?


I am working on the file "4 Data Processing.ipynb".
The first task is:
PCA versus LDA (50 points)
We have learned two different methods for feature extraction, PCA (unsupervised) and LDA (supervised).
Under what circumstances would PCA and LDA produce very different results?
Provide one example dataset in 2D, analyze it via PCA and LDA, and plot it with the PCA and LDA components.
You can use code from the scikit-learn library.
Answer
One simple case is several very tall and narrow clusters stacked horizontally.
PCA/LDA and will favor the vertical/horizontal direction, respectively.
To avoid curing via standardization, just having enough classes so that the total/individual set is sufficiently isotropic/anisotropic.
The difference could be explained:
PCA:
(1) unsupervised (no class label information)
(2) project data into dimensions that maximize variance.
LDA:
(1) supervised (with class label information)
(2) project data into dimensions to (a) maximize inter-class spread, (b) minimize intra-class spread
Can you write Python code for it?


I am working on the file "4 Stats for data analysis/Homework/1 test mean conf int/Test Mean conf int.ipynb".
The first task is:
<b>
Вспомним формулу доверительного интервала для среднего нормально распределённой случайной величины с дисперсией σ2:
При σ=1 какой нужен объём выборки, чтобы на уровне доверия 95% оценить среднее с точностью ±0.1?
Can you write Python code for it?

from scipy import stats
np.ceil((stats.norm.ppf(1-0.05/2) / 0.1)**2)


I am working on the file "4 Stats for data analysis/Homework/1 test mean conf int/Test Mean conf int.ipynb".
The first task is:
На данных из предыдущего вопроса постройте 95% доверительный интервал для средней годовой смертности по всем южным городам. Чему равна его верхняя граница? Округлите ответ до 4 знаков после десятичной точки.
Can you write Python code for it?

water_data_south = water_data[water_data.location == 'South']
mort_mean_south = water_data_south['mortality'].mean()
print('Mean south mortality: %f' % mort_mean_south)

mort_mean_south_std = water_data_south['mortality'].std() / np.sqrt(water_data_south['mortality'].shape[0])
print('Mortality south 95%% interval: %s' %  str(_tconfint_generic(mort_mean_south, mort_mean_south_std,
                                                                   water_data_south['mortality'].shape[0] - 1,
                                                                   0.05, 'two-sided')))


I am working on the file "4 Stats for data analysis/Homework/1 test mean conf int/Test Mean conf int.ipynb".
The first task is:
Пересекаются ли 95% доверительные интервалы для средней жёсткости воды в северных и южных городах?
Can you write Python code for it?

hardness_mean_south = water_data_south['hardness'].mean()
print('Mean south hardness: %f' % hardness_mean_south)

hardness_mean_north = water_data_north['hardness'].mean()
print('Mean north hardness: %f' % hardness_mean_north)

hardness_mean_south_std = water_data_south['hardness'].std() / np.sqrt(water_data_south['hardness'].shape[0])
print('Hardness south 95%% interval: %s' %  str(_tconfint_generic(hardness_mean_south, hardness_mean_south_std,
                                                                   water_data_south['hardness'].shape[0] - 1,
                                                                   0.05, 'two-sided')))

hardness_mean_north_std = water_data_north['hardness'].std() / np.sqrt(water_data_north['hardness'].shape[0])
print('Hardness north 95%% interval: %s' %  str(_tconfint_generic(hardness_mean_north, hardness_mean_north_std,
                                                                   water_data_north['hardness'].shape[0] - 1,
                                                                   0.05, 'two-sided')))


I am working on the file "4 Stats for data analysis/Homework/1 test mean conf int/Test Mean conf int.ipynb".
The first task is:
Постройте 95% доверительный интервал для средней годовой смертности в больших городах. Чему равна его нижняя граница? Округлите ответ до 4 знаков после десятичной точки.
Can you write Python code for it?

mort_mean = water_data['mortality'].mean()
print('Mean mortality: %f' % mort_mean)

from statsmodels.stats.weightstats import _tconfint_generic

mort_mean_std = water_data['mortality'].std() / np.sqrt(water_data['mortality'].shape[0])
print('Mortality 95%% interval: %s' %  str(_tconfint_generic(mort_mean, mort_mean_std, water_data['mortality'].shape[0] - 1,
                                                         0.05, 'two-sided')))


I am working on the file "4 Stats for data analysis/Homework/10 test multiple hypothesis testing/Test Multiple hypothesis testing.ipynb".
The first task is:
Классификатор C4.5 и три его модификации: с оптимизацией гиперпараметра m, гиперпараметра cf и с одновременной оптимизацией обоих гиперпараметров. Эти четыре классификатора сравнивались на 14 наборах данных. На каждом датасете был посчитан AUC каждого классификатора. Данные записаны в файле:
AUCs.txt
Используя критерий знаковых рангов, проведите попарное сравнение каждого классификатора с каждым. Выберите два классификатора, различие между которыми наиболее статистически значимо.
Can you write Python code for it?

aucs = pd.read_csv('AUCs.txt', delimiter='\t')
aucs.info()

aucs.columns = [u'Dataset', u'C4.5', u'C4.5+m', u'C4.5+cf', u'C4.5+m+cf']
aucs

w_stat = pd.DataFrame(columns=['Model 1', 'Model 2', 'Wilcoxon stat', 'p-value'])
k = 0
for i, j in combinations([1, 2, 3, 4], 2):
    w_stat.loc[k, 'Model 1'], w_stat.loc[k, 'Model 2'] = aucs.columns[i], aucs.columns[j]
    w_stat.loc[k, 'Wilcoxon stat'], w_stat.loc[k, 'p-value'] = stats.wilcoxon(aucs.iloc[:, i], aucs.iloc[:, j])
    k += 1

w_stat

top_diff_idx = w_stat.loc[:, 'p-value'].idxmin()
print('Two classifiers with the highest significance difference: %s & %s' % (w_stat.loc[top_diff_idx, 'Model 1'],
      w_stat.loc[top_diff_idx, 'Model 2']))


I am working on the file "4 Stats for data analysis/Homework/11 test regression/Test Regression.ipynb".
The first task is:
В разных признаках пропуски возникают по разным причинам и должны обрабатываться по-разному.
Например, в признаке agefm пропуски стоят только там, где evermarr=0, то есть, они соответствуют женщинам, никогда не выходившим замуж. Таким образом, для этого признака NaN соответствует значению "не применимо".
В подобных случаях, когда признак x1 на части объектов в принципе не может принимать никакие значения, рекомендуется поступать так:
создать новый бинарный признак
x2={1,0,x1='не применимо',иначе;
заменить "не применимо" в x1 на произвольную константу c, которая среди других значений x1 не встречается.
Теперь, когда мы построим регрессию на оба признака и получим модель вида
y=β0+β1x1+β2x2,
на тех объектах, где x1 было измерено, регрессионное уравнение примет вид
y=β0+β1x,
а там, где x1 было "не применимо", получится
y=β0+β1c+β2.
Выбор c влияет только на значение и интерпретацию β2, но не β1.
Давайте используем этот метод для обработки пропусков в agefm и heduc.
Создайте признак nevermarr, равный единице там, где в agefm пропуски.
Удалите признак evermarr — в сумме с nevermarr он даёт константу, значит, в нашей матрице X будет мультиколлинеарность.
Замените NaN в признаке agefm на cagefm=0.
У объектов, где nevermarr = 1, замените NaN в признаке heduc на cheduc1=−1 (ноль использовать нельзя, так как он уже встречается у некоторых объектов выборки).
Сколько осталось пропущенных значений в признаке heduc?
Can you write Python code for it?

botswana['nevermarr'] = [1 if botswana.loc[i, 'evermarr'] == 0 else 0 for i in range(botswana.shape[0])]
botswana.head()

np.unique(botswana.agefm[botswana.agefm.notnull()].values)

botswana.agefm[botswana.agefm.isnull()] = 0
botswana.head()

del botswana['evermarr']

botswana.heduc[botswana.heduc.isnull() & botswana.nevermarr.values == 1] = -1
botswana.head()

botswana.info()

botswana.heduc.isnull().value_counts()


I am working on the file "4 Stats for data analysis/Homework/11 test regression/Test Regression.ipynb".
The first task is:
Давайте проанализируем данные опроса 4361 женщин из Ботсваны:
botswana.tsv
О каждой из них мы знаем:
сколько детей она родила (признак ceb)
возраст (age)
длительность получения образования (educ)
религиозная принадлежность (religion)
идеальное, по её мнению, количество детей в семье (idlnchld)
была ли она когда-нибудь замужем (evermarr)
возраст первого замужества (agefm)
длительность получения образования мужем (heduc)
знает ли она о методах контрацепции (knowmeth)
использует ли она методы контрацепции (usemeth)
живёт ли она в городе (urban)
есть ли у неё электричество, радио, телевизор и велосипед (electric, radio, tv, bicycle)
Давайте научимся оценивать количество детей ceb по остальным признакам.
Загрузите данные и внимательно изучите их. Сколько разных значений принимает признак religion?
Can you write Python code for it?

botswana = pd.read_csv('botswana.tsv', delimiter='\t')
botswana.info()

botswana.columns
botswana.head()

print(botswana.religion.value_counts())


I am working on the file "4 Stats for data analysis/Homework/11 test regression/Test Regression.ipynb".
The first task is:
Признак usemeth_noans значим по критерию Стьюдента, то есть, при его удалении модель значимо ухудшится. Но вообще-то отдельно его удалять нельзя: из-за того, что мы перекодировали пропуски в usemeth произвольно выбранным значением cusemeth=−1, удалять usemeth_noans и usemeth можно только вместе.
Удалите из текущей модели usemeth_noans и usemeth. Проверьте критерием Фишера гипотезу о том, что качество модели не ухудшилось. Введите номер первой значащей цифры в достигаемом уровне значимости (например, если вы получили 5.5×10−8, нужно ввести 8).
Если достигаемый уровень значимости получился маленький, верните удалённые признаки; если он достаточно велик, оставьте модель без usemeth и usemeth_noans.
Can you write Python code for it?

formula3 = 'ceb ~ age + educ + idlnchld + knowmeth + agefm + heduc + urban + electric + bicycle \
+ nevermarr + idlnchld_noans + heduc_noans'
formula3

reg_m5 = smf.ols(formula3, data=botswana)
fitted_m5 = reg_m5.fit()
print(fitted_m5.summary())

print('F=%f, p=%.40f, k1=%f' % reg_m4.fit().compare_f_test(reg_m5.fit()))


I am working on the file "4 Stats for data analysis/Homework/11 test regression/Test Regression.ipynb".
The first task is:
Проверьте критерием Бройша-Пагана гомоскедастичность ошибки в построенной модели. Выполняется ли она?
Если ошибка гетероскедастична, перенастройте модель, сделав поправку Уайта типа HC1.
Can you write Python code for it?

print('Breusch-Pagan test: p=%f' % sms.het_breuschpagan(fitted_m.resid, fitted_m.model.exog)[1])

reg_m2 = smf.ols(formula, data=botswana)
fitted_m2 = reg_m2.fit(cov_type='HC1')
print(fitted_m2.summary())


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Биномиальное распределение (точный способ)
Can you write Python code for it?

def zconfint_binom(n, p, alpha=0.05):
    q = 1 - p
    m = n * p
    var = n * p * q
    z_stat = stats.norm.ppf(1 - alpha / 2)
    l_bound = int(round(m - z_stat * np.sqrt(var)))
    u_bound = int(round(m + z_stat * np.sqrt(var)))
    return (l_bound, u_bound)

#confidence intervals estimation
lim_bal_0_s = np.sort(lim_bal_0.values)
lim_bal_1_s = np.sort(lim_bal_1.values)

l_ind_0, u_ind_0 = zconfint_binom(len(lim_bal_0_s), 0.5)
l_ind_1, u_ind_1 = zconfint_binom(len(lim_bal_1_s), 0.5)

print('Conf. int. median LIMIT_BAL for default = 0: [%d, %d]' % (lim_bal_0_s[l_ind_0], lim_bal_0_s[u_ind_0]))
print('Conf. int. median LIMIT_BAL for default = 1: [%d, %d]' % (lim_bal_1_s[l_ind_1], lim_bal_1_s[u_ind_1]))


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Бутстрэп (приближённый способ)
Can you write Python code for it?

def get_bootstrap_samples(data, n_samples):
    indices = np.random.randint(0, len(data), (n_samples, len(data)))
    samples = data[indices]
    return samples

def stat_intervals(stat, alpha):
    boundaries = np.percentile(stat, [100 * alpha / 2., 100 * (1 - alpha / 2.)])
    return boundaries

#confidence intervals estimation
np.random.seed(0)

lim_bal_0_bs_med = map(np.median, get_bootstrap_samples(lim_bal_0.values, 100))
lim_bal_1_bs_med = map(np.median, get_bootstrap_samples(lim_bal_1.values, 100))

bnd_0 = stat_intervals(lim_bal_0_bs_med, 0.05)
bnd_1 = stat_intervals(lim_bal_1_bs_med, 0.05)

print('Conf. int. median LIMIT_BAL for default = 0: [%d, %d]' % (bnd_0[0], bnd_0[1]))
print('Conf. int. median LIMIT_BAL for default = 1: [%d, %d]' % (bnd_1[0], bnd_1[1]))


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
В основном берут кредиты люди с BS и MS. Принципиальная разница между распределниями тех, кто вернул, и тех, кто нет, не наблюдается.
Проверим гипотезу о том, что образование не влияет на то, вернет ли человек долг.
Построим график долей возврата кредитов.
Can you write Python code for it?

#samples of EDUCATION for default = 0/1
edu = credit_story.EDUCATION
edu_0 = credit_story.EDUCATION.loc[credit_story.default == 0]
edu_1 = credit_story.EDUCATION.loc[credit_story.default == 1]

#proportions of credit returns
edu_prop = np.empty( (len(np.unique(edu_0)), 2) )
for i, ed_val in enumerate(np.unique(edu)):
    edu_prop[i, 0] = ed_val
    edu_prop[i, 1] = edu_0[edu_0 == ed_val].shape[0] / (edu_0[edu_0 == ed_val].shape[0] + edu_1[edu_1 == ed_val].shape[0])

for ed_val, prop in edu_prop:
    print('Prop. of def.=0 for education=%d: %.2f' % (ed_val, prop))

#histogram of EDUCATION proportions distribution
_ = plt.figure(1, figsize=(7,7))
_ = plt.title('Proportion histogram of EDUCATION')
_ = plt.plot(edu_prop[:,0], edu_prop[:,1])
_ = plt.axis([0, 6, 0.5, 1])


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Воспользуемя доверительным интервалом на основе нормального распределения.
Can you write Python code for it?

#confident intervals for proportions
conf_int_0 = proportion_confint(np.where(sex_0 == 1)[0].shape[0], sex_0.shape[0])
conf_int_1 = proportion_confint(np.where(sex_1 == 1)[0].shape[0], sex_1.shape[0])

print('Conf. int. for prop. of men or default = 0: [%.4f, %.4f]' % conf_int_0)
print('Conf. int. for prop. of men or default = 1: [%.4f, %.4f]' % conf_int_1)


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Гипотеза о равенстве гендерного распределния отвергается. Данный результат явлется и практически значимым, т.к. 6% существенная величина для банков.
<b>
3. Образование (EDUCATION).
EDUCATION: образование (0 = доктор, 1 = магистр; 2 = бакалавр; 3 = выпускник школы; 4 = начальное образование; 5 = прочее; 6 = нет данных)
Can you write Python code for it?

#histogram of EDUCATION distribution
_ = plt.figure(1, figsize=(15,7))

_ = plt.subplot(121)
_ = plt.title('Histogram of EDUCATION')
_ = plt.hist(credit_story['EDUCATION'], edgecolor='k')

_ = plt.subplot(122)
_ = plt.title('Histogram of EDUCATION for default = 0 / 1')
_ = plt.hist(credit_story.EDUCATION.loc[credit_story.default == 0], edgecolor='k', label='Default = 0')
_ = plt.hist(credit_story.EDUCATION.loc[credit_story.default == 1], edgecolor='k', label='Default = 1')
_ = plt.legend()


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Гипотеза о том, что величина имеет равномерное распределение отвергается с очень высокой долей вероятности.
Результат имеет практическую значимость для выдачи кредита, доли возврата значимо отличаются для разных уровней образования.
4. Семейное положение (MARRIAGE).
MARRIAGE: (0 = отказываюсь отвечать; 1 = замужем/женат; 2 = холост; 3 = нет данных).
Can you write Python code for it?

#histogram of MARRIAGE distribution
_ = plt.figure(1, figsize=(15,7))

_ = plt.subplot(121)
_ = plt.title('Histogram of MARRIAGE')
_ = plt.hist(credit_story['MARRIAGE'], edgecolor='k')

_ = plt.subplot(122)
_ = plt.title('Histogram of MARRIAGE for default = 0 / 1')
_ = plt.hist(credit_story.MARRIAGE.loc[credit_story.default == 0], edgecolor='k', label='Default = 0')
_ = plt.hist(credit_story.MARRIAGE.loc[credit_story.default == 1], edgecolor='k', label='Default = 1')
_ = plt.legend()


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Доли невозвратов среди женатых и холостых людей различается примерно на 10%.
Проверим, как связан семейный статус с индикатором дефолта.
Can you write Python code for it?

#samples of MARRIAGE for default = 0/1
mar = credit_story.MARRIAGE
mar_0 = credit_story.MARRIAGE.loc[credit_story.default == 0]
mar_1 = credit_story.MARRIAGE.loc[credit_story.default == 1]

print('Value counts for marriage:')
mar.value_counts()


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Доля небольших невозвратных кредитов заметно выше. Большие суммы люди стремятся возвращать.
Проверим гипотезу о равенстве медианных значений кредитного лимита с помощью подходящей интервальной оценки.

H0: медианные значения равны
H1: медианные значения отличаются
Can you write Python code for it?

#samples of LIMIT_BAL for default = 0/1
lim_bal_0 = credit_story.LIMIT_BAL.loc[credit_story.default == 0]
lim_bal_1 = credit_story.LIMIT_BAL.loc[credit_story.default == 1]

print('Median LIMIT_BAL for default = 0: %.0f' % lim_bal_0.median())
print('Median LIMIT_BAL for default = 1: %.0f' % lim_bal_1.median())


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Женщины более склонны к невозврату кредита, чем мужчины.
Проверим гипотезу о том, что гендерный состав группы людей вернувших и не вернувших кредит отличается.
Построим доверительные интервалы для доли вернувших, не вернувших и для их разности.
Can you write Python code for it?

#samples of SEX for default = 0/1
sex_0 = credit_story.SEX.loc[credit_story.default == 0].values
sex_1 = credit_story.SEX.loc[credit_story.default == 1].values

#proportions of men to women in samples
m_to_w_0 = np.where(sex_0 == 1)[0].shape[0] / sex_0.shape[0]
m_to_w_1 = np.where(sex_1 == 1)[0].shape[0] / sex_1.shape[0]

print('Prop. of men for default = 0: %.4f' % m_to_w_0)
print('Prop. of men for default = 1: %.4f' % m_to_w_1)


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Используем статистический Z-критерий для доли для двух независимых выборок.

H0: гендерный состав не отличается
H1: гендерный состав отличается
Can you write Python code for it?

def proportions_diff_z_stat_ind(sample1, sample2):
    n1 = len(sample1)
    n2 = len(sample2)

    p1 = float(sum(sample1)) / n1
    p2 = float(sum(sample2)) / n2
    P = float(p1*n1 + p2*n2) / (n1 + n2)

    return (p1 - p2) / np.sqrt(P * (1 - P) * (1. / n1 + 1. / n2))

def proportions_diff_z_test(z_stat, alternative='two-sided'):
    if alternative not in ('two-sided', 'less', 'greater'):
        raise ValueError("alternative not recognized\n"
                         "should be 'two-sided', 'less' or 'greater'")

    if alternative == 'two-sided':
        return 2 * (1 - stats.norm.cdf(np.abs(z_stat)))

    if alternative == 'less':
        return stats.norm.cdf(z_stat)

    if alternative == 'greater':
        return 1 - stats.norm.cdf(z_stat)

print('p-value: %.15f' % proportions_diff_z_test(proportions_diff_z_stat_ind(np.abs(sex_0-2), np.abs(sex_1-2))))


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Как видно из интервальной оценки, интервальные оценки для медианы пересекаются.
Построим интервальную оценку для разности медиан.
Can you write Python code for it?

print('Difference between medians: %f' % (np.median(age_0) - np.median(age_1)))

delta_median_scores = map(lambda x: x[1] - x[0], zip(age_0_bs_med, age_1_bs_med))

stat_int = stat_intervals(delta_median_scores, 0.05)
print('95%% conf. int. for the difference between medians: [%f, %f]' % (stat_int[0], stat_int[1]))


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Молодые люди склонные не возвращать кредиты. Доля тех, кто взял кредит в 20 лет и вернул его, меньше доли оных, которые не вернули кредит.
Проверим гипотезу о равенстве медианных значений кредитного лимита с помощью подходящей интервальной оценки.

H0: медианные значения равны
H1: медианные значения отличаются
Can you write Python code for it?

#samples of AGE for default = 0/1
age_0 = credit_story.AGE.loc[credit_story.default == 0]
age_1 = credit_story.AGE.loc[credit_story.default == 1]

print('Median AGE for default = 0: %.0f' % age_0.median())
print('Median AGE for default = 1: %.0f' % age_1.median())


I am working on the file "4 Stats for data analysis/Homework/12 project credit scoring/Credit scoring.ipynb".
The first task is:
Составим таблицу сопряжённости для признаков default и MARRIAGE.
Can you write Python code for it?

conf_table = np.empty( (len(mar.value_counts()), 2) )
for i in range(len(mar.value_counts())):
    conf_table[i, 0] = len(mar_0.loc[mar_0 == i])
    conf_table[i, 1] = len(mar_1.loc[mar_1 == i])
conf_table


I am working on the file "4 Stats for data analysis/Homework/13 project churn analysis/Churn analysis.ipynb".
The first task is:
Вы прослушали большой курс и к текущему моменту обладете достаточными знаниями, чтобы попытаться самостоятельно выбрать нужный метод / инструмент / статистический критерий и сделать правильное заключение.
В этой части задания вам нужно будет самостоятельно решить, с помощью каких методов можно провести анализ эффективности удержания (churn) с помощью раличных методов (treatment = 0, treatment = 2) относительно контрольной группы пользователей (treatment = 1).
Что можно сказать об этих двух методах (treatment = 0, treatment = 2)? Одинаковы ли они с точки зрения эффективности? Каким бы методом вы бы посоветовали воспользоваться компании?
Не забудьте про поправку на множественную проверку! И не пользуйтесь односторонними альтернативами, поскольку вы не знаете, к каким действительно последствиям приводят тестируемые методы (treatment = 0, treatment = 2)!
Can you write Python code for it?

pt_states = pd.pivot_table(churn, values='account_length', index='state', columns=['treatment', 'churn'],
                               aggfunc=len, fill_value=0)
pt_states

#proportions of churn = 1 for each treatment
props_churn = np.array([pt_states.loc[:, treat].loc[:, True].sum() /
                        (pt_states.loc[:, treat].loc[:, False].sum() + pt_states.loc[:, treat].loc[:, True].sum())
                        for treat in range(3)])
for i, prop in enumerate(props_churn):
    print('Treatment #%d prop. of churn = 1: %.4f' % (i, prop))

#wilson intervals for proportions
wilson_ints = np.array([proportion_confint(pt_states.loc[:, treat].loc[:, True].sum(),
                                           (pt_states.loc[:, treat].loc[:, False].sum() + pt_states.loc[:, treat].loc[:, True].sum()),
                                           method='wilson') for treat in range(3)])
for i, wils in enumerate(wilson_ints):
    print('Treatment #%d prop. of churn = 1: %s' % (i, str(wils)))

def proportions_confint_diff_ind(sample1, sample2, alpha = 0.05):
    z = stats.norm.ppf(1 - alpha / 2.)
    l1 = sample1.loc[:, False].sum() + sample1.loc[:, True].sum()
    l2 = sample2.loc[:, False].sum() + sample2.loc[:, True].sum()
    p1 = sample1.loc[:, True].sum() / l1
    p2 = sample2.loc[:, True].sum() / l2

    left_boundary = (p1 - p2) - z * np.sqrt(p1 * (1 - p1)/ l1 + p2 * (1 - p2)/ l2)
    right_boundary = (p1 - p2) + z * np.sqrt(p1 * (1 - p1)/ l1 + p2 * (1 - p2)/ l2)

    return (left_boundary, right_boundary)

def proportions_diff_z_stat_ind(sample1, sample2):
    l1 = sample1.loc[:, False].sum() + sample1.loc[:, True].sum()
    l2 = sample2.loc[:, False].sum() + sample2.loc[:, True].sum()
    p1 = sample1.loc[:, True].sum() / l1
    p2 = sample2.loc[:, True].sum() / l2
    P = (p1*l1 + p2*l2) / (l1 + l2)

    return (p1 - p2) / np.sqrt(P * (1 - P) * (1. / l1 + 1. / l2))

def proportions_diff_z_test(z_stat, alternative = 'two-sided'):
    if alternative not in ('two-sided', 'less',...


I am working on the file "4 Stats for data analysis/Homework/13 project churn analysis/Churn analysis.ipynb".
The first task is:
Давайте попробуем применить полученные знания о разных видах корреляции и ее применимости на практике.
Рассмотрим пару признаков day_calls и mes_estim. Посчитайте корреляцию Пирсона между этими признаками на всех данных, ее значимость.
Can you write Python code for it?

print('Pearson correlation: %.4f' % stats.pearsonr(churn.day_calls, churn.mes_estim)[0])
print('P-value: %.4f' % stats.pearsonr(churn.day_calls, churn.mes_estim)[1])


I am working on the file "4 Stats for data analysis/Homework/13 project churn analysis/Churn analysis.ipynb".
The first task is:
Давайте рассмотрим всех пользователей из контрольной группы (treatment = 1). Для таких пользователей мы хотим проверить гипотезу о том, что штат абонента не влияет на то, перестанет ли абонент пользоваться услугами оператора.
Для этого мы воспользуемся критерием хи-квадрат. Построим таблицы сопряженности между каждой из всех 1275 возможных неупорядоченных пар штатов и значением признака churn. Для каждой такой таблицы 2x2 применить критерий хи-квадрат можно с помощью функции scipy.stats.chi2_contingency(subtable, correction=False).
Заметьте, что, например, (AZ, HI) и (HI, AZ) — это одна и та же пара. Обязательно выставим correction=False (о том, что это значит мы узнаем далее).
Сколько достигаемых уровней значимости оказались меньше, чем α=0.05?
Can you write Python code for it?

pt_states = pd.pivot_table(churn.loc[churn.treatment == 1], values='treatment', index='state', columns='churn',
                               aggfunc=len, fill_value=0)
pt_states

pt_states.index

chi2_val = pd.DataFrame(columns=['state 1', 'state 2', 'chi2_stat', 'p-value'])
i = 0
for st1, st2 in combinations(pt_states.index, 2):
    chi2_table = pt_states.loc[[st1, st2],:]
    chi2_val.loc[i, 'state 1'] = st1
    chi2_val.loc[i, 'state 2'] = st2
    chi2_stat = stats.chi2_contingency(chi2_table.values, correction=False)
    chi2_val.loc[i, 'chi2_stat'] = chi2_stat[0]
    chi2_val.loc[i, 'p-value'] = chi2_stat[1]
    i += 1

pd.set_option('display.max_rows', None)
chi2_val

sign_lvls_num = len(chi2_val.loc[chi2_val['p-value'] <= 0.05, :])
print('Num of significance lvls below 0.05: %d' % sign_lvls_num)


I am working on the file "4 Stats for data analysis/Homework/13 project churn analysis/Churn analysis.ipynb".
The first task is:
Еще раз рассмотрим пару признаков day_calls и mes_estim. Посчитайте корреляцию Спирмена между этими признаками на всех данных, ее значимость.
Can you write Python code for it?

print('Spearman correlation: %.4f' % stats.spearmanr(churn.day_calls, churn.mes_estim)[0])
print('P-value: %.4f' % stats.spearmanr(churn.day_calls, churn.mes_estim)[1])

_ = plt.figure(figsize=(15,6))
_ = plt.subplot(121)
_ = sc.stats.probplot(churn.day_calls, dist="norm", plot=plt)
_ = plt.subplot(122)
_ = sc.stats.probplot(churn.mes_estim, dist="norm", plot=plt)

_ = plt.scatter(churn.day_calls, churn.mes_estim, alpha=0.2)
_ = plt.xlabel('day_calls')
_ = plt.ylabel('mes_estim')


I am working on the file "4 Stats for data analysis/Homework/14 test AB browser test/AB browser test.ipynb".
The first task is:
Для каждого браузера в каждой из двух групп (control и exp) посчитаем долю запросов, в которых пользователь не кликнул ни разу. Это можно сделать, поделив сумму значений n_nonclk_queries на сумму значений n_queries. Умножив это значение на 100, получим процент некликнутых запросов, который можно легче проинтерпретировать.
Can you write Python code for it?

browsers_nonclk_q_exp = ab_data.loc[exp].groupby(['browser']).sum().loc[:,'n_nonclk_queries']
browsers_clk_q_exp = ab_data.loc[exp].groupby(['browser']).sum().loc[:,'n_queries']
browsers_nonclk_q_prop_exp = browsers_nonclk_q_exp / browsers_clk_q_exp

browsers_nonclk_q_ctrl = ab_data.loc[ctrl].groupby(['browser']).sum().loc[:,'n_nonclk_queries']
browsers_clk_q_ctrl = ab_data.loc[ctrl].groupby(['browser']).sum().loc[:,'n_queries']
browsers_nonclk_q_prop_ctrl = browsers_nonclk_q_ctrl / browsers_clk_q_ctrl

print('Control / experimental groups')
for br in browsers:
    print('Browser #%d' % br),
    print(browsers_nonclk_q_prop_ctrl.loc[browsers_nonclk_q_prop_ctrl.index == br].values),
    print('/'),
    print(browsers_nonclk_q_prop_exp.loc[browsers_nonclk_q_prop_ctrl.index == br].values)


I am working on the file "4 Stats for data analysis/Homework/14 test AB browser test/AB browser test.ipynb".
The first task is:
Одним из возможных аналогов t-критерия, которым можно воспрользоваться, является тест Манна-Уитни. На достаточно обширном классе распределений он является асимптотически более эффективным, чем t-критерий, и при этом не требует параметрических предположений о характере распределения.
Разделим выборку на две части, соответствующие control и exp группам. Преобразуем данные к виду, чтобы каждому пользователю соответствовало суммарное значение его кликов. С помощью критерия Манна-Уитни проверим гипотезу о равенстве средних.
Can you write Python code for it?

users_nclicks_exp = ab_data.loc[exp].groupby(['userID', 'browser']).sum().loc[:,'n_clicks']
users_nclicks_ctrl = ab_data.loc[ctrl].groupby(['userID', 'browser']).sum().loc[:,'n_clicks']

users_nclicks_exp.head()
users_nclicks_ctrl.head()

stats.mannwhitneyu(users_nclicks_exp, users_nclicks_ctrl, alternative='two-sided')


I am working on the file "4 Stats for data analysis/Homework/14 test AB browser test/AB browser test.ipynb".
The first task is:
Основная метрика, на которой мы сосредоточимся в этой работе, — это количество пользовательских кликов на web-странице в зависимости от тестируемого изменения этой страницы.
Посчитаем, насколько в группе exp больше пользовательских кликов по сравнению с группой control в процентах от числа кликов в контрольной группе.
Can you write Python code for it?

#number of people in exp and control groups
ab_data.slot.value_counts()


I am working on the file "4 Stats for data analysis/Homework/14 test AB browser test/AB browser test.ipynb".
The first task is:
Поскольку данных достаточно много (порядка полумиллиона уникальных пользователей), отличие в несколько процентов может быть не только практически значимым, но и значимым статистически. Последнее утверждение нуждается в дополнительной проверке.
Can you write Python code for it?

_ = plt.figure(figsize=(15,5))
_ = plt.subplot(121)
_ = plt.hist(ab_data.n_clicks.loc[exp], bins=100)
_ = plt.title('Experiment group')
_ = plt.subplot(122)
_ = plt.hist(ab_data.n_clicks.loc[ctrl], bins=100)
_ = plt.title('Control group')


I am working on the file "4 Stats for data analysis/Homework/14 test AB browser test/AB browser test.ipynb".
The first task is:
Проверим, для какого из браузеров наиболее сильно выражено отличие между количеством кликов в контрольной и экспериментальной группах.
Для этого применим для каждого из срезов (по каждому из уникальных значений столбца browser) критерий Манна-Уитни между control и exp группами и сделаем поправку Холма-Бонферрони на множественную проверку с α=0.05.
Can you write Python code for it?

browsers_nclicks_exp = ab_data.loc[exp].groupby(['browser', 'userID']).sum().loc[:,'n_clicks']
browsers_nclicks_ctrl = ab_data.loc[ctrl].groupby(['browser', 'userID']).sum().loc[:,'n_clicks']

browsers_nclicks_exp.head()
browsers_nclicks_ctrl.head()

#Unique browsers
browsers = np.unique(ab_data.browser)
print('Unique browsers numbers: ' + str(browsers))
print('')

print('Mann-Whitney rank test without multipletest')
mw_p = np.empty(browsers.shape[0])
for i, br in enumerate(browsers):
    print('Browser #%d: ' % br),
    _, mw_p[i] = stats.mannwhitneyu(browsers_nclicks_exp.loc[br, :], browsers_nclicks_ctrl.loc[br, :], alternative='two-sided')
    print('p-value = %.4f' % mw_p[i])

print('')
print('Mann-Whitney rank test with multipletest')
_, mw_p_corr, _, _ = multipletests(mw_p, alpha = 0.05, method = 'holm')
for i, br in enumerate(browsers):
    print('Browser #%d: ' % br),
    print('p-value = %.4f' % mw_p_corr[i])


I am working on the file "4 Stats for data analysis/Homework/15 project genom cancer/Genom cancer.ipynb".
The first task is:
Для того, чтобы использовать двухвыборочный критерий Стьюдента, убедимся, что распределения в выборках существенно не отличаются от нормальных, применив критерий Шапиро-Уилка.
Can you write Python code for it?

#Shapiro-Wilk test for samples
print('Shapiro-Wilk test for samples')

sw_normal = gen_normal.iloc[:,2:].apply(stats.shapiro, axis=0)
sw_normal_p = [p for _, p in sw_normal]
_, sw_normal_p_corr, _, _ = multipletests(sw_normal_p, method='fdr_bh')

sw_neoplasia = gen_neoplasia.iloc[:,2:].apply(stats.shapiro, axis=0)
sw_neoplasia_p = [p for _, p in sw_neoplasia]
_, sw_neoplasia_p_corr, _, _ = multipletests(sw_neoplasia_p, method='fdr_bh')

sw_cancer = gen_cancer.iloc[:,2:].apply(stats.shapiro, axis=0)
sw_cancer_p = [p for _, p in sw_cancer]
_, sw_cancer_p_corr, _, _ = multipletests(sw_cancer_p, method='fdr_bh')

print('Mean corrected p-value for "normal": %.4f' % sw_normal_p_corr.mean())
print('Mean corrected p-value for "early neoplasia": %.4f' % sw_neoplasia_p_corr.mean())
print('Mean corrected p-value for "cancer": %.4f' % sw_cancer_p_corr.mean())


I am working on the file "4 Stats for data analysis/Homework/15 project genom cancer/Genom cancer.ipynb".
The first task is:
Так как среднее значение p-value >> 0.05, то будем применять критерий Стьюдента.
Can you write Python code for it?

tt_ind_normal_neoplasia = stats.ttest_ind(gen_normal.iloc[:,2:], gen_neoplasia.iloc[:,2:], equal_var = False)
tt_ind_normal_neoplasia_p = tt_ind_normal_neoplasia[1]

tt_ind_neoplasia_cancer = stats.ttest_ind(gen_neoplasia.iloc[:,2:], gen_cancer.iloc[:,2:], equal_var = False)
tt_ind_neoplasia_cancer_p = tt_ind_neoplasia_cancer[1]

tt_ind_normal_neoplasia_p_5 = tt_ind_normal_neoplasia_p[np.where(tt_ind_normal_neoplasia_p < 0.05)].shape[0]
tt_ind_neoplasia_cancer_p_5 = tt_ind_neoplasia_cancer_p[np.where(tt_ind_neoplasia_cancer_p < 0.05)].shape[0]

print('Normal vs neoplasia samples p-values number below 0.05: %d' % tt_ind_normal_neoplasia_p_5)
print('Neoplasia vs cancer samples p-values number below 0.05: %d' % tt_ind_neoplasia_cancer_p_5)

with open('answer1.txt', 'w') as fout:
    fout.write(str(tt_ind_normal_neoplasia_p_5))

with open('answer2.txt', 'w') as fout:
    fout.write(str(tt_ind_neoplasia_cancer_p_5))


I am working on the file "4 Stats for data analysis/Homework/15 project genom cancer/Genom cancer.ipynb".
The first task is:
Часть 2: поправка методом Холма
Для этой части задания нам понадобится модуль multitest из statsmodels.
В этой части задания нужно будет применить поправку Холма для получившихся двух наборов достигаемых уровней значимости из предыдущей части. Обратим внимание, что поскольку мы будем делать поправку для каждого из двух наборов p-value отдельно, то проблема, связанная с множественной проверкой останется.
Для того, чтобы ее устранить, достаточно воспользоваться поправкой Бонферрони, то есть использовать уровень значимости 0.05 / 2 вместо 0.05 для дальнейшего уточнения значений p-value c помощью метода Холма.
В качестве ответа к этому заданию требуется ввести количество значимых отличий в каждой группе после того, как произведена коррекция Холма-Бонферрони. Причем это число нужно ввести с учетом практической значимости: посчитать для каждого значимого изменения fold change и выписать в ответ число таких значимых изменений, абсолютное значение fold change которых больше, чем 1.5.
Обратим внимание, что

применять поправку на множественную проверку нужно ко всем значениям достигаемых уровней значимости, а не только для тех, которые меньше значения уровня доверия;
при использовании поправки на уровне значимости 0.025 меняются значения достигаемого уровня значимости, но не меняется значение уровня доверия (то есть для отбора значимых изменений скорректированные значения уровня значимости нужно сравнивать с порогом 0.025, а не 0.05)!
Can you write Python code for it?

#Holm correction
_, tt_ind_normal_neoplasia_p_corr, _, _ = multipletests(tt_ind_normal_neoplasia_p, method='holm')
_, tt_ind_neoplasia_cancer_p_corr, _, _ = multipletests(tt_ind_neoplasia_cancer_p, method='holm')

#Bonferroni correction
p_corr = np.array([tt_ind_normal_neoplasia_p_corr, tt_ind_neoplasia_cancer_p_corr])
_, p_corr_bonf, _, _ = multipletests(p_corr, is_sorted=True, method='bonferroni')

p_corr_bonf_normal_neoplasia_p_5 = p_corr_bonf[0][np.where(p_corr_bonf[0] < 0.05)].shape[0]
p_corr_bonf_neoplasia_cancer_p_5 = p_corr_bonf[1][np.where(p_corr_bonf[1] < 0.05)].shape[0]

print('Normal vs neoplasia samples p-values number below 0.05: %d' % p_corr_bonf_normal_neoplasia_p_5)
print('Neoplasia vs cancer samples p-values number below 0.05: %d' % p_corr_bonf_neoplasia_cancer_p_5)

def fold_change(C, T, limit=1.5):
    '''
    C - control sample
    T - treatment sample
    '''
    if T >= C:
        fc_stat = T / C
    else:
        fc_stat = -C / T

    return (np.abs(fc_stat) > limit), fc_stat

#Normal vs neoplasia samples
gen_p_corr_bonf_normal_p_5 = gen_normal.iloc[:,2:].iloc[:, np.where(p_corr_bonf[0] < 0.05)[0]]
gen_p_corr_bonf_neoplasia0_p_5 = gen_neoplasia.iloc[:,2:].iloc[:, np.where(p_corr_bonf[0] < 0.05)[0]]

fc_corr_bonf_normal_neoplasia_p_5 = 0
for norm, neopl in zip(gen_p_corr_bonf_normal_p_5.mean(), gen_p_corr_bonf_neoplasia0_p_5.mean()):
    accept, _ = fold_change(norm, neopl)
    if accept: fc_corr_bonf_normal_neoplasia_p_5 += 1

#Neoplasia vs cancer samples
gen_p_corr_bonf_neoplasia1_p_5 = gen_neoplasia.iloc[:,2:].iloc[:, np.where(p_corr_bonf[1] < 0.05)[0]]
gen_p_corr_bonf_cancer_p_5 = gen_cancer.iloc[:,2:].iloc[:, np.where(p_corr_bonf[1] < 0.05)[0]]

fc_corr_bonf_neoplasia_cancer_p_5 = 0
for neopl, canc in zip(gen_p_corr_bonf_neoplasia1_p_5.mean(), gen_p_corr_bonf_cancer_p_5.mean()):
    accept, _ = fold_change(neopl, canc)
    if accept: fc_corr_bonf_neoplasia_cancer_p_5 += 1

print('Normal vs neoplasia samples fold change above 1.5: %d' %...


I am working on the file "4 Stats for data analysis/Homework/15 project genom cancer/Genom cancer.ipynb".
The first task is:
Часть 3: поправка методом Бенджамини-Хохберга
Данная часть задания аналогична второй части за исключением того, что нужно будет использовать метод Бенджамини-Хохберга.
Обратим внимание, что методы коррекции, которые контролируют FDR, допускает больше ошибок первого рода и имеют большую мощность, чем методы, контролирующие FWER. Большая мощность означает, что эти методы будут совершать меньше ошибок второго рода (то есть будут лучше улавливать отклонения от H0, когда они есть, и будут чаще отклонять H0, когда отличий нет).
В качестве ответа к этому заданию требуется ввести количество значимых отличий в каждой группе после того, как произведена коррекция Бенджамини-Хохберга, причем так же, как и во второй части, считать только такие отличия, у которых abs(fold change) > 1.5.
Can you write Python code for it?

#Benjamini-Hochberg correction
_, tt_ind_normal_neoplasia_p_corr, _, _ = multipletests(tt_ind_normal_neoplasia_p, method='fdr_bh')
_, tt_ind_neoplasia_cancer_p_corr, _, _ = multipletests(tt_ind_neoplasia_cancer_p, method='fdr_bh')

#Bonferroni correction
p_corr = np.array([tt_ind_normal_neoplasia_p_corr, tt_ind_neoplasia_cancer_p_corr])
_, p_corr_bonf, _, _ = multipletests(p_corr, is_sorted=True, method='bonferroni')

p_corr_bonf_normal_neoplasia_p_5 = p_corr_bonf[0][np.where(p_corr_bonf[0] < 0.05)].shape[0]
p_corr_bonf_neoplasia_cancer_p_5 = p_corr_bonf[1][np.where(p_corr_bonf[1] < 0.05)].shape[0]

print('Normal vs neoplasia samples p-values number below 0.05: %d' % p_corr_bonf_normal_neoplasia_p_5)
print('Neoplasia vs cancer samples p-values number below 0.05: %d' % p_corr_bonf_neoplasia_cancer_p_5)

#Normal vs neoplasia samples
gen_p_corr_bonf_normal_p_5 = gen_normal.iloc[:,2:].iloc[:, np.where(p_corr_bonf[0] < 0.05)[0]]
gen_p_corr_bonf_neoplasia0_p_5 = gen_neoplasia.iloc[:,2:].iloc[:, np.where(p_corr_bonf[0] < 0.05)[0]]

fc_corr_bonf_normal_neoplasia_p_5 = 0
for norm, neopl in zip(gen_p_corr_bonf_normal_p_5.mean(), gen_p_corr_bonf_neoplasia0_p_5.mean()):
    accept, _ = fold_change(norm, neopl)
    if accept: fc_corr_bonf_normal_neoplasia_p_5 += 1

#Neoplasia vs cancer samples
gen_p_corr_bonf_neoplasia1_p_5 = gen_neoplasia.iloc[:,2:].iloc[:, np.where(p_corr_bonf[1] < 0.05)[0]]
gen_p_corr_bonf_cancer_p_5 = gen_cancer.iloc[:,2:].iloc[:, np.where(p_corr_bonf[1] < 0.05)[0]]

fc_corr_bonf_neoplasia_cancer_p_5 = 0
for neopl, canc in zip(gen_p_corr_bonf_neoplasia1_p_5.mean(), gen_p_corr_bonf_cancer_p_5.mean()):
    accept, _ = fold_change(neopl, canc)
    if accept: fc_corr_bonf_neoplasia_cancer_p_5 += 1

print('Normal vs neoplasia samples fold change above 1.5: %d' % fc_corr_bonf_normal_neoplasia_p_5)
print('Neoplasia vs cancer samples fold change above 1.5: %d' % fc_corr_bonf_neoplasia_cancer_p_5)

with open('answer5.txt', 'w') as fout:
    fout.write(str(fc_corr_bonf_normal_neoplasia_p_5))

with...


I am working on the file "4 Stats for data analysis/Homework/2 test proportions conf int/Test Proportions conf int.ipynb".
The first task is:
Большая часть млекопитающих неспособны во взрослом возрасте переваривать лактозу, содержащуюся в молоке. У людей за расщепление лактозы отвечает фермент лактаза, кодируемый геном LCT. У людей с вариантом 13910T этого гена лактаза продолжает функционировать на протяжении всей жизни. Распределение этого варианта гена сильно варьируется в различных генетических популяциях.
Из 50 исследованных представителей народа майя вариант 13910T был обнаружен у одного. Постройте нормальный 95% доверительный интервал для доли носителей варианта 13910T в популяции майя. Чему равна его нижняя граница? Округлите ответ до 4 знаков после десятичной точки.
Can you write Python code for it?

size = 50
data_gen = np.zeros(size)
data_gen[0] = 1
data_gen

from statsmodels.stats.proportion import proportion_confint

normal_interval = proportion_confint(sum(data_gen), len(data_gen), method = 'normal')
print('Normal interval [%.4f, %.4f] with width %f' % (normal_interval[0],
                                                  normal_interval[1],
                                                  normal_interval[1] - normal_interval[0]))


I am working on the file "4 Stats for data analysis/Homework/2 test proportions conf int/Test Proportions conf int.ipynb".
The first task is:
В условиях предыдущей задачи постройте 95% доверительный интервал Уилсона для доли носителей варианта 13910T в популяции майя. Чему равна его нижняя граница? Округлите ответ до 4 знаков после десятичной точки.
Can you write Python code for it?

wilson_interval = proportion_confint(sum(data_gen), len(data_gen), method = 'wilson')
print('Wilson interval [%.4f, %.4f] with width %f' % (wilson_interval[0],
                                                  wilson_interval[1],
                                                  wilson_interval[1] - wilson_interval[0]))


I am working on the file "4 Stats for data analysis/Homework/2 test proportions conf int/Test Proportions conf int.ipynb".
The first task is:
Постройте график зависимости объёма выборки, необходимой для оценки для доли носителей гена 13910T с точностью ±0.01 на уровне доверия 95%, от неизвестного параметра p. Посмотрите, при каком значении p нужно больше всего испытуемых. Как вы думаете, насколько вероятно, что выборка, которую мы анализируем, взята из случайной величины с этим значением параметра?
Как бы вы не ответили на последний вопрос, рассмотреть объём выборки, необходимый при таком p, всё равно полезно — это даёт максимально пессимистичную оценку необходимого объёма выборки.
Какой объём выборки нужен в худшем случае, чтобы с помощью нормального интервала оценить долю носителей гена 13910T с точностью ±0.01 на уровне доверия 95%?
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

proportion = np.linspace(0, 1, 101)
proportion

n_samples = np.empty(proportion.shape)
for i, p in enumerate(proportion):
    n_samples[i] = int(np.ceil(samplesize_confint_proportion(p, 0.01)))
n_samples

plt.plot(proportion, n_samples);

n_samples[np.where(proportion == 0.5)]


I am working on the file "4 Stats for data analysis/Homework/2 test proportions conf int/Test Proportions conf int.ipynb".
The first task is:
Пусть в популяции майя действительно 2% носителей варианта 13910T, как в выборке, которую мы исследовали. Какой объём выборки нужен, чтобы с помощью нормального интервала оценить долю носителей гена 13910T с точностью ±0.01 на уровне доверия 95%?
Can you write Python code for it?

from statsmodels.stats.proportion import samplesize_confint_proportion

n_samples = int(np.ceil(samplesize_confint_proportion(data_gen.mean(), 0.01)))
n_samples


I am working on the file "4 Stats for data analysis/Homework/3 test bootstrap conf int/Test Bootstrap conf int.ipynb".
The first task is:
Величина, которую вы оценили в предыдущем вопросе, называется отношением шансов. Постройте для отношения шансов 95% доверительный интервал с помощью бутстрепа. Чему равна его нижняя граница? Округлите ответ до 4 знаков после десятичной точки.
Чтобы получить в точности такой же доверительный интервал, как у нас:
составьте векторы исходов в контрольной и тестовой выборках так, чтобы в начале шли все единицы, а потом все нули;
установите random seed=0;
сделайте по 1000 псевдовыборок из каждой группы пациентов с помощью функции get_bootstrap_samples.
Can you write Python code for it?

def get_bootstrap_samples(data, n_samples):
    indices = np.random.randint(0, len(data), (n_samples, len(data)))
    samples = data[indices]
    return samples

def stat_intervals(stat, alpha):
    boundaries = np.percentile(stat, [100 * alpha / 2., 100 * (1 - alpha / 2.)])
    return boundaries

np.random.seed(0)
odds_aspirin_data = np.array(map(odds, get_bootstrap_samples(aspirin_data, 1000)))
odds_placebo_data = np.array(map(odds, get_bootstrap_samples(placebo_data, 1000)))

print('95%% confidence interval for times decrease of infarction: %s' %
      str(stat_intervals(odds_placebo_data / odds_aspirin_data, 0.05)))


I am working on the file "4 Stats for data analysis/Homework/3 test bootstrap conf int/Test Bootstrap conf int.ipynb".
The first task is:
Продолжим анализировать данные эксперимента Гарвардской медицинской школы.
Для бернуллиевских случайных величин X∼Ber(p) часто вычисляют величину p1−p, которая называется шансами (odds). Чтобы оценить шансы по выборке, вместо p нужно подставить p^. Например, шансы инфаркта в контрольной группе, принимавшей плацебо, можно оценить как
Оцените, во сколько раз понижаются шансы инфаркта при регулярном приёме аспирина. Округлите ответ до четырёх знаков после десятичной точки.
Can you write Python code for it?

def odds(data):
    p = data.sum() / data.shape[0]
    return p / (1 - p)

odds_aspirin = odds(aspirin_data)
print('Odds aspirin: %.4f' % odds_aspirin)
odds_placebo = odds(placebo_data)
print('Odds aspirin: %.4f' % odds_placebo)

print('Times decrease of infarction: %.4f' % (odds_placebo / odds_aspirin))


I am working on the file "4 Stats for data analysis/Homework/4 test hypothesis testing/Test Hypothesis testing.ipynb".
The first task is:
<b>
The Wage Tract — заповедник в округе Тома, Джорджия, США, деревья в котором не затронуты деятельностью человека со времён первых поселенцев. Для участка заповедника размером 200х200 м имеется информация о координатах сосен (sn — координата в направлении север-юг, we — в направлении запад-восток, обе от 0 до 200).
pines.txt
Проверим, можно ли пространственное распределение сосен считать равномерным, или они растут кластерами.
Загрузите данные, поделите участок на 5х5 одинаковых квадратов размера 40x40 м, посчитайте количество сосен в каждом квадрате (чтобы получить такой же результат, как у нас, используйте функцию scipy.stats.binned_statistic_2d).
Если сосны действительно растут равномерно, какое среднее ожидаемое количество сосен в каждом квадрате? В правильном ответе два знака после десятичной точки.
Can you write Python code for it?

pines_data = pd.read_table('pines.txt')
pines_data.describe()
pines_data.head()

sns.pairplot(pines_data, size=4);

sn_num, we_num = 5, 5

trees_bins = stats.binned_statistic_2d(pines_data.sn, pines_data.we, None, statistic='count', bins=[sn_num, we_num])

trees_squares_num = trees_bins.statistic
trees_squares_num
trees_bins.x_edge
trees_bins.y_edge

mean_trees_num = np.sum(trees_squares_num) / 25
print(mean_trees_num)


I am working on the file "4 Stats for data analysis/Homework/4 test hypothesis testing/Test Hypothesis testing.ipynb".
The first task is:
<b>
Представим теперь, что в другой ресторанной сети только 22 из 50 работников испытывают существенный стресс. Гипотеза о том, что 22/50 соответствует 75% по всей популяции, методом, который вы использовали в предыдущей задаче, отвергается. Чем это может объясняться? Выберите все возможные варианты.
Can you write Python code for it?

print('p-value: %.10f' % stats.binom_test(22, 50, prob))


I am working on the file "4 Stats for data analysis/Homework/5 test student tests/Test Student tests.ipynb".
The first task is:
<b>
Имеются данные о стоимости и размерах 53940 бриллиантов:
diamonds.txt
Отделите 25% случайных наблюдений в тестовую выборку с помощью функции sklearn.cross_validation.train_test_split (зафиксируйте random state = 1). На обучающей выборке настройте две регрессионные модели:
линейную регрессию с помощью LinearRegression без параметров
случайный лес с помощью RandomForestRegressor с random_state=1.
Какая из моделей лучше предсказывает цену бриллиантов? Сделайте предсказания на тестовой выборке, посчитайте модули отклонений предсказаний от истинных цен. Проверьте гипотезу об одинаковом среднем качестве предсказаний, вычислите достигаемый уровень значимости. Отвергается ли гипотеза об одинаковом качестве моделей против двусторонней альтернативы на уровне значимости α=0.05?
Can you write Python code for it?

diamonds = pd.read_table('diamonds.txt')
diamonds.describe()
diamonds.head()
diamonds.info()

diamonds.columns

X_diam = diamonds.drop(['price'], axis=1)
X_diam.shape
y_diam = diamonds.loc[:, diamonds.columns == 'price']
np.ravel(y_diam).shape

X_diam_train, X_diam_test, y_diam_train, y_diam_test = train_test_split(X_diam, y_diam, random_state=1)

clf_lr = LinearRegression()
clf_lr.fit(X_diam_train, y_diam_train)

pred_price_lr = clf_lr.predict(X_diam_test)

pred_price_lr_sub = pred_price_lr - y_diam_test
pred_price_lr_sub.describe()

clf_rf = RandomForestRegressor(random_state=1)
clf_rf.fit(X_diam_train, y_diam_train.values.ravel())

pred_price_rf = clf_rf.predict(X_diam_test)

pred_price_rf_sub = pred_price_rf[:, np.newaxis] - y_diam_test
pred_price_rf_sub.describe()

stats.ttest_rel(np.abs(pred_price_lr_sub), np.abs(pred_price_rf_sub))

print "95%% confidence interval: [%f, %f]" % DescrStatsW(np.abs(pred_price_lr_sub) - np.abs(pred_price_rf_sub)).tconfint_mean()


I am working on the file "4 Stats for data analysis/Homework/5 test student tests/Test Student tests.ipynb".
The first task is:
Уровень кальция в крови здоровых молодых женщин равен в среднем 9.5 милиграммам на децилитр и имеет характерное стандартное отклонение 0.4 мг/дл. В сельской больнице Гватемалы для 160 здоровых беременных женщин при первом обращении для ведения беременности был измерен уровень кальция; среднее значение составило 9.57 мг/дл. Можно ли утверждать, что средний уровень кальция в этой популяции отличается от 9.5?
Посчитайте достигаемый уровень значимости. Поскольку известны только среднее и дисперсия, а не сама выборка, нельзя использовать стандартные функции критериев — нужно реализовать формулу достигаемого уровня значимости самостоятельно.
Округлите ответ до четырёх знаков после десятичной точки.
Can you write Python code for it?

exp_calc = 9.5
st_dev_calc = 0.4

num = 160
mean_calc = 9.57

def z_test(mean_val, exp_val, st_dev, num):
    standard_error = st_dev / np.sqrt(num)
    return (mean_val - exp_val) / standard_error

z_stat_calc = z_test(mean_calc, exp_calc, st_dev_calc, num)
z_stat_calc

def p_val_double(z_stat):
    return 2 * (1 - stats.norm.cdf(z_stat))

print('p-value: %.4f' % p_val_double(z_stat_calc))


I am working on the file "4 Stats for data analysis/Homework/6 test parametric tests/Test Parametric tests.ipynb".
The first task is:
<b>
Ежегодно более 200000 людей по всему миру сдают стандартизированный экзамен GMAT при поступлении на программы MBA. Средний результат составляет 525 баллов, стандартное отклонение — 100 баллов.
Сто студентов закончили специальные подготовительные курсы и сдали экзамен. Средний полученный ими балл — 541.4. Проверьте гипотезу о неэффективности программы против односторонней альтернативы о том, что программа работает. Отвергается ли на уровне значимости 0.05 нулевая гипотеза? Введите достигаемый уровень значимости, округлённый до 4 знаков после десятичной точки.
Can you write Python code for it?

avg_mark = 541.4
avg_mark_data = np.ones(100) * avg_mark

exp_mark = 525.
st_dev = 100.

def z_test(mean_val, exp_val, st_dev, num):
    standard_error = st_dev / np.sqrt(num)
    return (mean_val - exp_val) / standard_error

def p_val_greater(z_stat):
    return 1 - stats.norm.cdf(z_stat)

z_stat = z_test(avg_mark, exp_mark, st_dev, (len(avg_mark_data)))
z_stat

p_val = p_val_greater(z_stat)
p_val

avg_mark2 = 541.5

z_stat2 = z_test(avg_mark2, exp_mark, st_dev, (len(avg_mark_data)))
z_stat2

p_val2 = p_val_greater(z_stat2)
p_val2


I am working on the file "4 Stats for data analysis/Homework/6 test parametric tests/Test Parametric tests.ipynb".
The first task is:
В одном из выпусков программы "Разрушители легенд" проверялось, действительно ли заразительна зевота. В эксперименте участвовало 50 испытуемых, проходивших собеседование на программу. Каждый из них разговаривал с рекрутером; в конце 34 из 50 бесед рекрутер зевал. Затем испытуемых просили подождать решения рекрутера в соседней пустой комнате.
Во время ожидания 10 из 34 испытуемых экспериментальной группы и 4 из 16 испытуемых контрольной начали зевать. Таким образом, разница в доле зевающих людей в этих двух группах составила примерно 4.4%. Ведущие заключили, что миф о заразительности зевоты подтверждён.
Можно ли утверждать, что доли зевающих в контрольной и экспериментальной группах отличаются статистически значимо? Посчитайте достигаемый уровень значимости при альтернативе заразительности зевоты, округлите до четырёх знаков после десятичной точки.
Can you write Python code for it?

data_exp = np.array( [1 if i < 10 else 0 for i in range(34)] )
data_ctrl = np.array( [1 if i < 4 else 0 for i in range(16)] )

print('Mean experimental value: %.4f' % data_exp.mean())
print('Mean control value: %.4f' % data_ctrl.mean())

conf_interval_banner_exp = proportion_confint(np.sum(data_exp), len(data_exp), method = 'wilson')
conf_interval_banner_ctrl = proportion_confint(np.sum(data_ctrl), len(data_ctrl), method = 'wilson')

print('95%% confidence interval for exp group: [%f, %f]' % conf_interval_banner_exp)
print('95%% confidence interval for a ctrl group: [%f, %f]' % conf_interval_banner_ctrl)

def proportions_diff_confint_ind(sample1, sample2, alpha = 0.05):
    z = stats.norm.ppf(1 - alpha / 2.)

    p1 = float(sum(sample1)) / len(sample1)
    p2 = float(sum(sample2)) / len(sample2)

    left_boundary = (p1 - p2) - z * np.sqrt(p1 * (1 - p1)/ len(sample1) + p2 * (1 - p2)/ len(sample2))
    right_boundary = (p1 - p2) + z * np.sqrt(p1 * (1 - p1)/ len(sample1) + p2 * (1 - p2)/ len(sample2))

    return (left_boundary, right_boundary)

def proportions_diff_z_stat_ind(sample1, sample2):
    n1 = len(sample1)
    n2 = len(sample2)

    p1 = float(sum(sample1)) / n1
    p2 = float(sum(sample2)) / n2
    P = float(p1*n1 + p2*n2) / (n1 + n2)

    return (p1 - p2) / np.sqrt(P * (1 - P) * (1. / n1 + 1. / n2))

def proportions_diff_z_test(z_stat, alternative='two-sided'):
    if alternative not in ('two-sided', 'less', 'greater'):
        raise ValueError("alternative not recognized\n"
                         "should be 'two-sided', 'less' or 'greater'")

    if alternative == 'two-sided':
        return 2 * (1 - stats.norm.cdf(np.abs(z_stat)))

    if alternative == 'less':
        return stats.norm.cdf(z_stat)

    if alternative == 'greater':
        return 1 - stats.norm.cdf(z_stat)

print('95%% confidence interval for a difference: [%.4f, %.4f]' % proportions_diff_confint_ind(data_exp, data_ctrl))

print('p-value: %.4f' %...


I am working on the file "4 Stats for data analysis/Homework/7 test non-parametric tests/Test Non-parametric test.ipynb".
The first task is:
<b>
В ходе исследования влияния лесозаготовки на биоразнообразие лесов острова Борнео собраны данные о количестве видов деревьев в 12 лесах, где вырубка не ведётся:
22,22,15,13,19,19,18,20,21,13,13,15,
и в 9 лесах, где идёт вырубка:
17,18,18,15,12,4,14,15,10.
Проверьте гипотезу о равенстве среднего количества видов в двух типах лесов против односторонней альтернативы о снижении биоразнообразия в вырубаемых лесах. Используйте ранговый критерий. Чему равен достигаемый уровень значимости? Округлите до четырёх знаков после десятичной точки.
Can you write Python code for it?

forest_not_cut = np.array( [22, 22, 15, 13, 19, 19, 18, 20, 21, 13, 13, 15] )
forest_not_cut.mean()
forest_cut = np.array( [17, 18, 18, 15, 12, 4, 14, 15, 10] )
forest_cut.mean()

stats.mannwhitneyu(forest_not_cut, forest_cut, alternative='greater')