I am working on the file "MasteringML_wSkLearn/02a_Classification.ipynb".
The first task is:
R squre Score is quite high which says there is a large proportion of the variance in the data
Multiple Linear Regression
Can you write Python code for it?

# in this there are two features of pizza - diameter and no of toppings

X = [[6,2],[8,1],[10, 0],[14,2],[18, 0]]
y = [[7],[9],[13],[17.5],[18]]
X_test = [[8,2],[9,0],[11,2],[16,2],[12, 0]]
y_test = [[11],[8.5],[15],[18],[11]]

lReg2 = LinearRegression()
lReg2.fit(X, y)

y_pred = lReg2.predict(X_test)

print("Training Score:", lReg2.score(X_test, y_test))
print("Test Score:", lReg2.score(X_test, y_test))

for i, yp in enumerate(y_pred):
    print("predicted : ",yp, " Actual : ", y_test[i])


I am working on the file "MasteringML_wSkLearn/02a_Classification.ipynb".
The first task is:
Simple Linear Regression
Can you write Python code for it?

lReg = LinearRegression()

# fitting the data
lReg.fit(X, y)

# predicting the data for 12" pizza
lReg.predict([[12]])

y_pred = lReg.predict(X)

# plotting the new point in the plot

X = [[6], [8], [10], [14], [18]]
y = [[7], [9], [13], [17.5], [18]]

plt.figure()
plt.title('Pizza price plotted against diameter')
plt.xlabel('Diameter in inches')
plt.ylabel('Price in dollars')
plt.plot(X, y, 'k.')
plt.plot([12], lReg.predict([[12]]), c='r', marker='.')
plt.axis([0, 25, 0, 25])
plt.grid(True)

x1 = [[0], [25]]
y1 = lReg.predict(x1)
plt.plot(x1, y1, 'g')
plt.show()

# priting model coeff and intercept
print("Co-efficient : ", lReg.coef_)
print("Intercept : ", lReg.intercept_)


I am working on the file "MasteringML_wSkLearn/02a_Classification.ipynb".
The first task is:
Variance
Variance is a measure of how far a set of values is spread out. If all of the numbers
in the set are equal, the variance of the set is zero. A small variance indicates that the
numbers are near the mean of the set, while a set containing numbers that are far
from the mean and each other will have a large variance. Variance can be calculated
using the following equation:
$$ VAR = \frac{\sum_{i=1}^{n} (x_i - \overline{x})^2 }{n-1} $$
Can you write Python code for it?

print("X: ", X)
print ("mean X: ", np.mean(X))

var = np.sum(np.square(X - np.mean(X)))/(len(X)-1)
print("Variance: ", var)

print("Variance: ",np.var(X, ddof=1))


I am working on the file "MasteringML_wSkLearn/02a_Classification.ipynb".
The first task is:
We know that simple linear regression follows the below equation :
 $$ y = \alpha + \beta{x} $$
where $$ \beta = \frac{cov(X,y)}{var(x)}  $$
Can you write Python code for it?

# let's get the beta
beta = covar/var
alpha = np.mean(y) - beta*np.mean(X)
print("beta b (co-efficient)= ", beta)
print("alpha a (intercept)= ", alpha)

x2 = [[11], [18]]
print(lReg.predict(x2))


I am working on the file "MasteringML_wSkLearn/02b_Classification.ipynb".
The first task is:
Fitting models with gradient descent
There are two varieties of gradient descent that are distinguished by the number
of training instances that are used to update the model parameters in each training
iteration. Batch gradient descent, which is sometimes called only gradient descent,
uses all of the training instances to update the model parameters in each iteration.
Stochastic Gradient Descent (SGD), in contrast, updates the parameters using
only a single training instance in each iteration. The training instance is usually
selected randomly. Stochastic gradient descent is often preferred to optimize cost
functions when there are hundreds of thousands of training instances or more, as
it will converge more quickly than batch gradient descent. Batch gradient descent
is a deterministic algorithm, and will produce the same parameter values given the
same training set.
As a stochastic algorithm, SGD can produce different parameter
estimates each time it is run. SGD may not minimize the cost function as well as
gradient descent because it uses only single training instances to update the weights.
Its approximation is often close enough, particularly for convex cost functions such
as residual sum of squares.
Can you write Python code for it?

boston = load_boston()

# splitting the data
X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target, test_size=0.2, random_state=27)

#
X_scaler = StandardScaler()
y_scaler = StandardScaler()
X_train = X_scaler.fit_transform(X_train)
y_train = y_scaler.fit_transform(y_train)
X_test = X_scaler.transform(X_test)
y_test = y_scaler.transform(y_test)

regressor = SGDRegressor(loss='squared_loss')
scores = cross_val_score(regressor, X_train, y_train, cv=5)
print ('Cross validation r-squared scores:', scores)
print ('Average cross validation r-squared score:', np.mean(scores))
regressor.fit_transform(X_train, y_train)
print ('Test set r-squared score', regressor.score(X_test, y_test))


I am working on the file "MasteringML_wSkLearn/03_Feature_Extraction_&_Preprocessing.ipynb".
The first task is:
As we can see both sentences are having same meaning but their feature vectors have no elements in common. Let's use the lexical analysis on the data
Can you write Python code for it?

lemmatizer = WordNetLemmatizer()
print (lemmatizer.lemmatize('gathering', 'v'))
print (lemmatizer.lemmatize('gathering', 'n'))


I am working on the file "MasteringML_wSkLearn/03_Feature_Extraction_&_Preprocessing.ipynb".
The first task is:
CountVectorizer
Can you write Python code for it?

corpus = [
    'UNC played Duke in basketball',
    'Duke lost the basketball game'
]

vectorizer = CountVectorizer()
print (vectorizer.fit_transform(corpus).todense())
print (vectorizer.vocabulary_)

# adding one more sentence in corpus

corpus = [
    'UNC played Duke in basketball',
    'Duke lost the basketball game',
    'This is Atul Singh'
]

vectorizer = CountVectorizer()
print (vectorizer.fit_transform(corpus).todense())
print (vectorizer.vocabulary_)

# checking the euclidean distance

# converting sentence into CountVectorizer
counts = vectorizer.fit_transform(corpus).todense()

print("1 & 2", euclidean_distances(counts[0], counts[1]))
print("2 & 3", euclidean_distances(counts[1], counts[2]))
print("1 & 3", euclidean_distances(counts[0], counts[2]))


I am working on the file "MasteringML_wSkLearn/03_Feature_Extraction_&_Preprocessing.ipynb".
The first task is:
DictVectorizer
Can you write Python code for it?

onehot_encoder = DictVectorizer()
instances = [
    {'city': 'New York'},
    {'city': 'San Francisco'},
    {'city': 'Chapel Hill'} ]

print (onehot_encoder.fit_transform(instances).toarray())


I am working on the file "MasteringML_wSkLearn/03_Feature_Extraction_&_Preprocessing.ipynb".
The first task is:
Stemming and Lemmatization
Lemmatization is the process of determining the lemma, or the morphological root, of an inflected word based on its context. Lemmas are the base forms of words that are used to key the word in a dictionary.
Stemming has a similar goal to lemmatization, but it does not attempt to produce the morphological roots of words. Instead, stemming removes all patterns of characters that appear to be affixes, resulting in a token that is not necessarily a valid word.
Lemmatization frequently requires a lexical resource, like WordNet, and the word's part of speech. Stemming
algorithms frequently use rules instead of lexical resources to produce stems and can
operate on any token, even without its context.
Can you write Python code for it?

corpus = [
    'He ate the sandwiches',
    'Every sandwich was eaten by him'
]

vectorizer = CountVectorizer(stop_words='english')  # added one option which remove the grammer words from corpus
print (vectorizer.fit_transform(corpus).todense())
print (vectorizer.vocabulary_)


I am working on the file "MasteringML_wSkLearn/03_Feature_Extraction_&_Preprocessing.ipynb".
The first task is:
Stop Word Filtering
Can you write Python code for it?

vectorizer = CountVectorizer(stop_words='english')  # added one option which remove the grammer words from corpus
print (vectorizer.fit_transform(corpus).todense())
print (vectorizer.vocabulary_)

print("1 & 2", euclidean_distances(counts[0], counts[1]))
print("2 & 3", euclidean_distances(counts[1], counts[2]))
print("1 & 3", euclidean_distances(counts[0], counts[2]))


I am working on the file "MasteringML_wSkLearn/03_Feature_Extraction_&_Preprocessing.ipynb".
The first task is:
The Porter stemmer cannot consider the inflected form's part of speech and returns gather for both documents:
Can you write Python code for it?

stemmer = PorterStemmer()
print (stemmer.stem('gathering'))

wordnet_tags = ['n', 'v']
corpus = [
'He ate the sandwiches',
'Every sandwich was eaten by him'
]
stemmer = PorterStemmer()
print ('Stemmed:', [[stemmer.stem(token) for token in word_tokenize(document)] for document in corpus])

def lemmatize(token, tag):
	if tag[0].lower() in ['n', 'v']:
		return lemmatizer.lemmatize(token, tag[0].lower())
	return token
lemmatizer = WordNetLemmatizer()

tagged_corpus = [pos_tag(word_tokenize(document)) for document in corpus]
print ('Lemmatized:', [[lemmatize(token, tag) for token, tag in document] for document in tagged_corpus])


I am working on the file "MasteringML_wSkLearn/04a_Logistic_Regression.ipynb".
The first task is:
Accuracy
Accuracy measures a fraction of the classifier's predictions that are correct.
Can you write Python code for it?


I am working on the file "MasteringML_wSkLearn/04a_Logistic_Regression.ipynb".
The first task is:
Calculating the F1 measure
$$ F1 = 2\frac{PR}{P + R} $$
The F1 measure penalizes classifiers with imbalanced precision and recall scores,
like the trivial classifier that always predicts the positive class. A model with perfect
precision and recall scores will achieve an F1 score of one. A model with a perfect
precision score and a recall score of zero will achieve an F1 score of zero. As for
precision and recall, scikit-learn provides a function to calculate the F1 score for
a set of predictions.
Can you write Python code for it?


I am working on the file "MasteringML_wSkLearn/04a_Logistic_Regression.ipynb".
The first task is:
From the ROC AUC plot, it is apparent that our classifier outperforms random
guessing; most of the plot area lies under its curve
Tuning models with grid search
Hyperparameters are parameters of the model that are not learned. For example,
hyperparameters of our logistic regression SMS classifier include the value of
the regularization term and thresholds used to remove words that appear too
frequently or infrequently. In scikit-learn, hyperparameters are set through the
model's constructor. In the previous examples, we did not set any arguments for
LogisticRegression(); we used the default values for all of the hyperparameters.
These default values are often a good start, but they may not produce the optimal
model.
Grid search is a common method to select the hyperparameter values
that produce the best model. Grid search takes a set of possible values for each
hyperparameter that should be tuned, and evaluates a model trained on each
element of the Cartesian product of the sets. That is, grid search is an exhaustive
search that trains and evaluates a model for each possible combination of the
hyperparameter values supplied by the developer. A disadvantage of grid search
is that it is computationally costly for even small sets of hyperparameter values.
Fortunately, it is an embarrassingly parallel problem; many models can easily be
trained and evaluated concurrently since no synchronization is required between
the processes. Let's use scikit-learn's GridSearchCV() function to find better
hyperparameter values:
Can you write Python code for it?

pipeline = Pipeline([
    ('vect', TfidfVectorizer(stop_words='english')),
    ('clf', LogisticRegression())
])

parameters = {
    'vect__max_df': (0.25, 0.5, 0.75),
    'vect__stop_words': ('english', None),
    'vect__max_features': (2500, 5000, 10000, None),
    'vect__ngram_range': ((1, 1), (1, 2)),
    'vect__use_idf': (True, False),
    'vect__norm': ('l1', 'l2'),
    'clf__penalty': ('l1', 'l2'),
    'clf__C': (0.01, 0.1, 1, 10),
}

grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='accuracy', cv=3)
grid_search.fit(X_train_raw, y_train)

print( 'Best score: %0.3f' % grid_search.best_score_)
print( 'Best parameters set:')

best_parameters = grid_search.best_estimator_.get_params()

for param_name in sorted(parameters.keys()):
    print( '\t%s: %r' % (param_name, best_parameters[param_name]))

predictions = grid_search.predict(X_test_raw)
predictions = [0 if i == 'ham' else 1 for i in predictions]

print( 'Accuracy:', accuracy_score(y_test, predictions))
print( 'Precision:', precision_score(y_test, predictions))
print( 'Recall:', recall_score(y_test, predictions))


I am working on the file "MasteringML_wSkLearn/04b_Logistic_Regression.ipynb".
The first task is:
Lets revise
Can you write Python code for it?

# reading the data
df = pd.read_csv("data/fertility_Diagnosis.txt", delimiter=',', header=None)
df.iloc[:4,0:9]

X_train, X_test, y_train, y_test = train_test_split(df.iloc[:,0:9], df[9], test_size=0.1)

pipeline = Pipeline([('clf', LogisticRegression())])

parameters = {
 'clf__penalty': ('l1', 'l2'),
    'clf__C': (0.01, 0.1, 1, 10)
    }

grid_search = GridSearchCV(pipeline, parameters, n_jobs=5, verbose=True, scoring='accuracy', cv = 5)
grid_search.fit(X_train, y_train)

print( 'Best score: %0.3f' % grid_search.best_score_)
print( 'Best parameters set:')

best_parameters = grid_search.best_estimator_.get_params()

for param_name in sorted(parameters.keys()):
    print( '\t%s: %r' % (param_name, best_parameters[param_name]))

y_pred = grid_search.predict(X_test)

#print((y_pred), (y_test))

y_test = [2 if x=='N' else 1 for x in y_test]
y_pred = [2 if x=='N' else 1 for x in y_pred]

#print((y_pred), (y_test))

print( 'Accuracy:', accuracy_score(y_test, y_pred))
print( 'Precision:', precision_score(y_test, y_pred))
print( 'Recall:', recall_score(y_test, y_pred))


I am working on the file "MasteringML_wSkLearn/04c_Logistic_Regression.ipynb".
The first task is:
hamming_loss
Can you write Python code for it?


I am working on the file "MasteringML_wSkLearn/04c_Logistic_Regression.ipynb".
The first task is:
jaccard_similarity_score
Can you write Python code for it?


I am working on the file "MasteringML_wSkLearn/05_Decision_Trees.ipynb".
The first task is:
Tree ensembles (RandomForestClassifier)
Ensemble learning methods combine a set of models to produce an estimator that
has better predictive performance than its individual components. A random forest
is a collection of decision trees that have been trained on randomly selected subsets
of the training instances and explanatory variables. Random forests usually make
predictions by returning the mode or mean of the predictions of their constituent
trees.
Random forests are less prone to overfitting than decision trees because no single
tree can learn from all of the instances and explanatory variables; no single tree can
memorize all of the noise in the representation
Can you write Python code for it?

pipeline = Pipeline([
('clf', RandomForestClassifier(criterion='entropy'))
])

parameters = {
'clf__n_estimators': (5, 10, 20, 50),
'clf__max_depth': (50, 150, 250),
'clf__min_samples_split': (1, 2, 3),
'clf__min_samples_leaf': (1, 2, 3)
}

grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='f1')
#grid_search.fit(X_train, y_train)


I am working on the file "MasteringML_wSkLearn/06_Clustering_with_K-Means.ipynb".
The first task is:
Clustering to learn features
Can you write Python code for it?

all_instance_filenames = []
all_instance_targets = []

for f in glob.glob('C:/Users/atul.singh/Downloads/cat_dog_test/test1/*.jpg'):
	target = 1 if 'cat' in f else 0
	all_instance_filenames.append(f)
	all_instance_targets.append(target)

surf_features = []
counter = 0

for f in all_instance_filenames:
	#print('Reading image:', f)
	image = mh.imread(f, as_grey=True)
	surf_features.append(surf.surf(image)[:, 5:])

train_len = int(len(all_instance_filenames) * .60)

X_train_surf_features = np.concatenate(surf_features[:train_len])
X_test_surf_feautres = np.concatenate(surf_features[train_len:])
y_train = all_instance_targets[:train_len]
y_test = all_instance_targets[train_len:]

n_clusters = 300
print ('Clustering', len(X_train_surf_features), 'features')
estimator = MiniBatchKMeans(n_clusters=n_clusters)
estimator.fit_transform(X_train_surf_features)

X_train = []
for instance in surf_features[:train_len]:
	clusters = estimator.predict(instance)
	features = np.bincount(clusters)
	if len(features) < n_clusters:
		features = np.append(features, np.zeros((1, n_clusterslen(features))))
		X_train.append(features)

X_test = []
for instance in surf_features[train_len:]:
	clusters = estimator.predict(instance)
	features = np.bincount(clusters)
	if len(features) < n_clusters:
		features = np.append(features, np.zeros((1, n_clusterslen(features))))
	X_test.append(features)

clf = LogisticRegression(C=0.001, penalty='l2')
clf.fit_transform(X_train, y_train)
predictions = clf.predict(X_test)
print(classification_report(y_test, predictions))
print('Precision: ', precision_score(y_test, predictions))
print('Recall: ', recall_score(y_test, predictions))
print('Accuracy: ', accuracy_score(y_test, predictions))


I am working on the file "MasteringML_wSkLearn/07_Dimensionality_Reduction_with_PCA.ipynb".
The first task is:
Eigenvectors and eigenvalues
An eigenvector of a matrix is a non-zero vector that satisfies the following equation:
$$ A \overrightarrow{v} = \lambda\overrightarrow{v} $$
Can you write Python code for it?

w, v = np.linalg.eig(np.array([[1, -2], [2, -3]]))
print("eigen vector : ", w)
print("eigen values : ",v)


I am working on the file "MasteringML_wSkLearn/07_Dimensionality_Reduction_with_PCA.ipynb".
The first task is:
The first eigenvector
has the greatest eigenvalue and is the first principal component. We will build a
transformation matrix in which each column of the matrix is the eigenvector for
a principal component. If we were reducing a five-dimensional data set to three
dimensions, we would build a matrix with three columns
Can you write Python code for it?


I am working on the file "MasteringML_wSkLearn/07_Dimensionality_Reduction_with_PCA.ipynb".
The first task is:
principal components are the eigenvectors of the data's covariance matrix ordered
by their eigenvalues.
Way1: The first technique requires calculating the covariance matrix of the
data. Since the covariance matrix will be square, we can calculate the eigenvectors
and eigenvalues
Way2: The
second technique uses singular value decomposition of the data matrix to find the
eigenvectors and square roots of the eigenvalues of the covariance matrix.
Can you write Python code for it?

cov = np.cov(x11, x21)
print(cov)

w, v = np.linalg.eig(cov)

print("eigen vector : ", w)
print("eigen values : ",v)


I am working on the file "MatPlotLib.ipynb".
The first task is:
Adjust the Axes
Can you write Python code for it?


I am working on the file "MatPlotLib.ipynb".
The first task is:
Bar Chart
Can you write Python code for it?

values = [12, 55, 4, 32, 14]
colors = ['r', 'g', 'b', 'c', 'm']
plt.bar(range(0,5), values, color= colors)
plt.show()


I am working on the file "MatPlotLib.ipynb".
The first task is:
Change Line Types and Colors
Can you write Python code for it?


I am working on the file "MatPlotLib.ipynb".
The first task is:
Histogram
Can you write Python code for it?

incomes = np.random.normal(27000, 15000, 10000)
plt.hist(incomes, 50)
plt.show()


I am working on the file "MatPlotLib.ipynb".
The first task is:
Labeling Axes and Adding a Legend
Can you write Python code for it?

axes = plt.axes()
axes.set_xlim([-5, 5])
axes.set_ylim([0, 1.0])
axes.set_xticks([-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5])
axes.set_yticks([0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])
axes.grid()
plt.xlabel('Greebles')
plt.ylabel('Probability')
plt.plot(x, norm.pdf(x), 'b-')
plt.plot(x, norm.pdf(x, 1.0, 0.5), 'r:')
plt.legend(['Sneetches', 'Gacks'], loc=4)
plt.show()


I am working on the file "MatPlotLib.ipynb".
The first task is:
Mutiple Plots on One Graph
Can you write Python code for it?

plt.plot(x, norm.pdf(x))
plt.plot(x, norm.pdf(x, 1.0, 0.5))
plt.show()


I am working on the file "MatPlotLib.ipynb".
The first task is:
Scatter Plot
Can you write Python code for it?

from pylab import randn

X = randn(500)
Y = randn(500)
plt.scatter(X,Y)
plt.show()


I am working on the file "MatPlotLib.ipynb".
The first task is:
XKCD Style :)
Can you write Python code for it?

plt.xkcd()

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
plt.xticks([])
plt.yticks([])
ax.set_ylim([-30, 10])

data = np.ones(100)
data[70:] -= np.arange(30)

plt.annotate(
    'THE DAY I REALIZED\nI COULD COOK BACON\nWHENEVER I WANTED',
    xy=(70, 1), arrowprops=dict(arrowstyle='->'), xytext=(15, -10))

plt.plot(data)

plt.xlabel('time')
plt.ylabel('my overall health')


I am working on the file "MatSE501/MatSE501-pycalphad.ipynb".
The first task is:
And phases:
Can you write Python code for it?


I am working on the file "MatSE501/MatSE501-pycalphad.ipynb".
The first task is:
Binary phase diagram
The equilibria along these individual energy curves can be built up to make a whole phase diagram.
Here we use binplot (binary plot) as a shortcut, which takes the same database, components, phases and conditions. However we would like our phase diagram to be over many temperatures and compositions, rather than a single point, so we have to express that in the conditions.
We will redefine our conditions to have the same kinds of conditions, however now the values in the parenthesis determine the (min, max, step) of our phase diagram calculations:
Note that this may take a minute or two. If you get a warning (in red), it is ok.
Can you write Python code for it?

from pycalphad import binplot

conditions = {
    v.P: 101325,
    v.T: (1000, 2000, 10), # (min, max, step)
    v.X('CR'): (0, 1, 0.05)
        }

binplot(db, components, phases, conditions)


I am working on the file "MatSE501/MatSE501-pycalphad.ipynb".
The first task is:
Conditions
In any thermodynamic calculation, $c+2$ conditions must be defined for $c$ components.
In pycalphad, we always calculate molar quantites, with the implicit condition that $n=1$, so all calculations must have $c+1$ conditions specified.
Each condition, pressure, temperature and composition are accessed via by prepending the type with a v:

Pressure: v.P in Pacals. Usually (but not always) thermodyanmic databases do not consider pressure and thus you use standard pressure, 101325 Pa by convention.
Temperature: v.T in Kelvin.
Composition: v.X('NI') as a mole fraction.

Sample equilibrium calculation
Can you write Python code for it?

from pycalphad import equilibrium
from pycalphad import variables as v

# Define components:
# Note: must have the correct capitialization.
# CALPHAD modelers usually use all-caps by convention.
# Also not that we include VA. Always do this.
components = ['CR', 'NI', 'VA']

# Define the phases:
phases = ['LIQUID', 'BCC_A2', 'FCC_A1']

# And the conditions:
# We want to see which phases are presnt when 10% Cr is in Ni
# This is a map of the variable to its value
conditions = {
    v.P: 101325,
    v.T: 500,
    v.X('CR'): 0.20
        }

# do the equlibrium calculation and store the result in the `eq` variable
eq = equilibrium(db, components, phases, conditions)


I am working on the file "MatSE501/MatSE501-pycalphad.ipynb".
The first task is:
If you got a warning in the above cell, that's ok.
Now we will look at what pycalphad calculated in the eq object. It is an xarray dataset.
xarray allows for the storage of values such as the stable phases, Gibbs energies, chemical potentials, as values over a grid of our conditions and phases given by the Gibbs phase rule.
If you want to learn more about xarray Datasets, see the xarray documentation.
Below you can see the dimensions (the grid of conditions) and the data variables (things that we calculated as a function of the different dimensions). The coordinates are just things we can index over, for convience.
Can you write Python code for it?


I am working on the file "MatSE501/MatSE501-pycalphad.ipynb".
The first task is:
Note that not all phases can be defined for every calculation. When all of the sublattices do not have at least one of the defined component defined, that phase may not be included. The SIGMA phase has only Cr definedin the second sublattice, and thus cannot be used when Cr is not a component.
Can you write Python code for it?


I am working on the file "MatSE501/MatSE501-pycalphad.ipynb".
The first task is:
Ternary (isothermal) phase diagram
For higher order equilibrium and other calculations, the procedure is mostly the same.
pycalphad can do equilibrium calculations of arbitrarily high-dimensioned systems (n-components, p-phases), but we are currently limited the visualizing in 2D.
We can plot an isothermal Gibbs triangle phase diagram using ternplot, the ternary version of binplot.
Can you write Python code for it?

from pycalphad import ternplot

tern_comps = ['CR', 'FE', 'NI', 'VA']
tern_phases = list(db.phases.keys())
conditions = {
    v.P: 101325,
    v.T: 1700, # (min, max, step)
    v.X('CR'): (0, 1, 0.025),
    v.X('NI'): (0, 1, 0.025),
        }

ternplot(db, tern_comps, tern_phases, conditions)


I am working on the file "MatSE501/MatSE501-pycalphad.ipynb".
The first task is:
Where the values of the chemical potential for Cr is -14676 J/mol and -19190 J/mol for Ni, as seen in their alignment with the coordinates.
Free energy curves
Now lets see if that chemical potential makes sense based on the free energy curves. The next section plots the free energy curves for all the phases we defined before.
Note that we are using the calculate function, rather than equilibrium.
calculate is slightly different than equilibrium in that it doesn't find the minimum energy, but rather samples all of the energies in the internal degrees of freedom. Hence, we do not have to specify any composition conditions and we specify temperature and pressure directly.
We'll plot the line between the chemical potentials we just calculated in red.
Can you write Python code for it?

%matplotlib inline
from pycalphad import calculate
from pycalphad.plot.utils import phase_legend
import matplotlib.pyplot as plt
import numpy as np

legend_handles, colorlist = phase_legend(sorted(phases))

fig = plt.figure()
ax = fig.gca()
ax.set_title('Cr-Ni 500K')
ax.set_ylabel('GM')
ax.set_xlabel('X(Cr)')
for phase in phases:
    result = calculate(db, components, phase, T=500, P=101325, output='GM')
    ax.scatter(result.X.sel(component='CR'), result.GM,
               marker='.', s=5, color=colorlist[phase.upper()])
ax.set_xlim((0, 1))
ax.plot([0, 1], [-19190, -14676], color='red', linewidth=3)
ax.legend(handles=legend_handles, loc='center left', bbox_to_anchor=(1, 0.6))


I am working on the file "MatSE501/MatSE501-pycalphad.ipynb".
The first task is:
You can see that the line between the chemical potentials we calculated is indeed tangent to the Gibbs free energy of the FCC phase.
Moving to higher temperature, we can see the same effect between two phases:
Can you write Python code for it?

T = 1400

# note that we can directly replace the variables with their values in the equilibrium function
eq_2 = equilibrium(db, components, phases, {v.P: 101325, v.T: T, v.X('CR'): 0.80})

fig = plt.figure()
ax = fig.gca()
ax.set_title('Cr-Ni {}K'.format(T))
ax.set_ylabel('GM')
ax.set_xlabel('X(Cr)')
for phase in phases:
    result = calculate(db, components, phase, T=T, P=101325, output='GM')
    ax.scatter(result.X.sel(component='CR'), result.GM,
               marker='.', s=5, color=colorlist[phase.upper()])
ax.set_xlim((0, 1))
# note that here we are indexing the chemical potentials by the selecting by *index* (isel)
ax.plot([1, 0], eq_2.MU.isel(P=0, T=0, X_CR=0), color='red', linewidth=3)
ax.legend(handles=legend_handles, loc='center left', bbox_to_anchor=(1, 0.6))


I am working on the file "MatSE501/MatSE501-pycalphad.ipynb".
The first task is:
You can see we calculated

NP: the phase fraction
GM: the molar Gibbs free energy
MU: the chemical potentials
X: The overall composition of each element in each phase (the constitution)
Y: The site fractions (compositions of each sublattice) in each phase
Phase: the name of the stable phase

Let's look at the values for the phase that was stable:
Note that the empty set of parenthesis means that there could have been a second phase there by Gibbs phase rule.
Can you write Python code for it?


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Additional description of the data
Can you write Python code for it?

describe(pwlaux['ML_LR'])

describe(pwlaux['lr'])

len(pwlaux)

np.sum(
    (pwlaux["AllWISE"] != pwlaux["AllWISE_input"]) |
    (pwlaux["objID"] != pwlaux["objID_input"])
      )

np.sum(
    (pwlaux["AllWISE"] != pwlaux["AllWISE_input"]) &
    (pwlaux["objID"] != pwlaux["objID_input"])
      )

np.sum(
    (pwl[cond_mlr].filled()["AllWISE"] != pwl[cond_mlr]["AllWISE_input"]) &
    (pwl[cond_mlr].filled()["objID"] == pwl[cond_mlr]["objID_input"])
      )

np.sum(
    (pwl[cond_mlr].filled()["AllWISE"] == pwl[cond_mlr]["AllWISE_input"]) &
    (pwl[cond_mlr].filled()["objID"] != pwl[cond_mlr]["objID_input"])
      )

for i in ["AllWISE", "AllWISE_input", "objID", "objID_input"]:
    print(i)
    most_common(pwl[cond_mlr][i].filled(), n=3)

Q_0_colour

np.sum(Q_0_colour)


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Analyse changes in the matches
Can you write Python code for it?


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Change the AllWISE_input that are '' to 'N/A'. That comes from a previous error with the fill value.
Can you write Python code for it?

pwl["AllWISE_input"][pwl["AllWISE_input"] == ""] = "N/A"


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Diagnostic columns
Can you write Python code for it?

pwl['match_code'] = 0

pwl['match_code'][np.isin(pwl["Source_Name"], pwl[cond_mlr & ~np.isnan(pwl['ML_LR'])][cond_diffmatch]["Source_Name"])] = 2

pwl['match_code'][np.isin(pwl["Source_Name"], pwl[cond_mlr & ~np.isnan(pwl['ML_LR'])][cond_match]["Source_Name"])]=1

pwl['match_code'][np.isin(pwl["Source_Name"], pwl[cond_mlr & ~np.isnan(pwl['ML_LR'])][cond_nomatch]["Source_Name"])] = 3

pwl['match_code'][np.isin(pwl["Source_Name"], pwl[cond_mlr & np.isnan(pwl['ML_LR'])][cond2_match]["Source_Name"])] = 4

pwl['match_code'][np.isin(pwl["Source_Name"], pwl[cond_mlr & np.isnan(pwl['ML_LR'])][cond2_nomatch]["Source_Name"])] = 5

for i in range(6):
    print(i, np.sum(pwl['match_code'] == i))


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Explore and repair data
Can you write Python code for it?

for col in pwl.colnames:
    fv = pwl[col].fill_value
    typ = pwl[col].dtype
    print(col, fv, typ)
    # Restore NaNs
    if fv == 1e+20:
        pwl[col][(pwl[col] == fv)] = np.nan
#     if (isinstance(fv, np.float64) and (fv != 1e+20)):
#         print(col, fv)
#         pwl[col].fill_value = 1e+20

pwl["colour"][(pwl["colour"] == 1e+20)] = np.nan

describe(pwl["colour"])


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Figure 4 of the paper: completeness and reliability
Can you write Python code for it?

q0 = np.sum(Q_0_colour)

def completeness(lr, threshold, q0):
    n = len(lr)
    lrt = lr[lr < threshold]
    return 1. - np.sum((q0 * lrt)/(q0 * lrt + (1 - q0)))/float(n)/q0

def reliability(lr, threshold, q0):
    n = len(lr)
    lrt = lr[lr > threshold]
    return 1. - np.sum((1. - q0)/(q0 * lrt + (1 - q0)))/float(n)/q0

completeness_v = np.vectorize(completeness, excluded=[0])
reliability_v = np.vectorize(reliability, excluded=[0])

pwl["lrt"] = pwl["lr"]
pwl["lrt"][np.isnan(pwl["lr"])] = 0

n_test = 100
threshold_mean = np.percentile(pwl["lrt"], 100*(1 - q0))

thresholds = np.arange(0., 10., 0.01)
thresholds_fine = np.arange(0.1, 2., 0.001)
completeness_t = completeness_v(pwl["lrt"], thresholds, q0)
reliability_t = reliability_v(pwl["lrt"], thresholds, q0)
average_t = (completeness_t + reliability_t)/2

completeness_t_fine = completeness_v(pwl["lrt"], thresholds_fine, q0)
reliability_t_fine = reliability_v(pwl["lrt"], thresholds_fine, q0)
average_t_fine = (completeness_t_fine + reliability_t_fine)/2

thresholds_fine[np.argmax(average_t_fine)]

thresholds_fine[np.argmin(np.abs(completeness_t_fine-reliability_t_fine))]

np.sum(pwl["lrt"] >= 0.358)

np.sum(pwl["lrt"] >= 0.639)

np.sum(pwl["lrt"] >= 0.639)

len(pwl)

len(pwl[pwl['ID_flag'] == 1])

n0 = np.sum((pwl["lrt"] >= 0.639) & (pwl['ID_flag'] == 1))
print(n0, n0/len(pwl[pwl['ID_flag'] == 1]))

n0 = np.sum((pwl["lrt"] >= 0.81268) & (pwl['ID_flag'] == 1))
print(n0, n0/len(pwl[pwl['ID_flag'] == 1]))

n0 = np.sum((pwl["ML_LR"] >= 0.81268) & (pwl['ID_flag'] == 1))
print(n0, n0/len(pwl[pwl['ID_flag'] == 1]))

np.sum(pwl["lrt"] >= 0.639)/len(pwl)

threshold_sel = 0.639

thresholds_fine = np.arange(0.0, 2., 0.001)
completeness_t_fine = completeness_v(pwl["lrt"], thresholds_fine, q0)
reliability_t_fine = reliability_v(pwl["lrt"], thresholds_fine, q0)

plt.rcParams["figure.figsize"] = (6.64, 6.64*0.74)
plt.rcParams["figure.dpi"] = 100
plt.rcParams['lines.linewidth'] = 1.75
plt.rcParams['lines.markersize'] = 8.0
plt.rcParams['lines.markeredgewidth'] =...


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
General configuration
Can you write Python code for it?

save_intermediate = True
plot_intermediate = True

idp = "idata/final_analysis_pdf_v1.0"

if not os.path.isdir(idp):
    os.makedirs(idp)


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Join data tables
Can you write Python code for it?

pwl = join(pwli, lofar_all[['Source_Name', 'AllWISE', 'objID', 'ML_LR',
                            'ID_flag', 'ID_name', 'ID_ra', 'ID_dec',
                            'LGZ_Size', 'LGZ_Width', 'LGZ_PA', 'LGZ_Assoc',
                            'LGZ_Assoc_Qual', 'LGZ_ID_Qual']],
           join_type='left',
           keys='Source_Name',
           uniq_col_name='{col_name}{table_name}',
           table_names=['', '_input'])

colour_limits = [0.0, 0.5, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 2.75, 3.0, 3.5, 4.0]

bin_list, centers, Q_0_colour, n_m, q_m = pickle.load(open("lofar_params.pckl", "rb"))


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Matched sources
Can you write Python code for it?

pwlaux_match = pwlaux[~np.isnan(pwlaux['ML_LR'])]

len(pwlaux_match)

cond_match = (
    ~np.isnan(pwlaux_match['lr']) &
    (pwlaux_match['lr'] >= threshold_sel) &
    (
        (pwlaux_match["AllWISE_input"] != "N/A") |
         ~np.isnan(pwlaux_match['objID_input'])
    ) &
    (
        (
            (pwlaux_match["AllWISE"] == pwlaux_match["AllWISE_input"]) &
            (pwlaux_match["objID"] == pwlaux_match["objID_input"]) &
            ~np.isnan(pwlaux_match["objID"]) &
            (pwlaux_match["AllWISE"] != "N/A")
        ) |
        (
            (pwlaux_match["AllWISE"] == pwlaux_match["AllWISE_input"]) &
            np.isnan(pwlaux_match["objID"])
        ) |
        (
            (pwlaux_match["AllWISE"] == "N/A") &
            (pwlaux_match["objID"] == pwlaux_match["objID_input"])
        )
    )

)
m_m = np.sum(cond_match)
print(m_m)

cond_diffmatch = (
    ~np.isnan(pwlaux_match['lr']) &
    (pwlaux_match['lr'] >= threshold_sel) &
    (
        (pwlaux_match["AllWISE_input"] != "N/A") |
         ~np.isnan(pwlaux_match['objID_input'])
    ) &
    (
        (
            (pwlaux_match["AllWISE"] != pwlaux_match["AllWISE_input"]) |
            (pwlaux_match["objID"] != pwlaux_match["objID_input"])
        )
    )

)
m_dm = np.sum(cond_diffmatch)
print(m_dm)

cond_nomatch = (
    np.isnan(pwlaux_match['lr']) |
    (pwlaux_match['lr'] < threshold_sel)
)
m_nm = np.sum(cond_nomatch)
print(m_nm)

m_nm + m_dm + m_m

217070+512+1021


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
N_LOFAR
Can you write Python code for it?

cond_mlr = (pwl['ID_flag'] == 1)

len(pwl)

np.sum(np.isnan(pwl["category"]))

np.sum(np.isnan(pwl["category"][cond_mlr]))

n_c, n_c_mlr = [], []
for i in np.unique(pwl["category"][~np.isnan(pwl["category"])]):
    n_c.append(np.sum((pwl["category"] == i)))
    n_c_mlr.append(np.sum((pwl["category"][cond_mlr] == i)))
    print(i, n_c[-1], n_c_mlr[-1])

total_n_c = np.sum(n_c)
total_n_c_mlr = np.sum(n_c_mlr)
print(total_n_c, total_n_c_mlr)
print(len(pwl)-np.sum(np.isnan(pwl["category"])), len(pwl)-np.sum(np.isnan(pwl["category"][cond_mlr])))
for i in range(16):
    print("{:2d} {:6d} {:6d} {:6.3f} {:6.3f} {:.2%}".format(i,
                                                     n_c[i],
                                                     n_c_mlr[i],
                                                     n_c[i]/total_n_c*100,
                                                     n_c_mlr[i]/total_n_c_mlr*100,
                                                     (n_c[i]-n_c_mlr[i])/n_c[i]
         ))

total_n_c


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Non-matched sources
Can you write Python code for it?

pwlaux_nomatch = pwlaux[np.isnan(pwlaux['ML_LR'])]

len(pwlaux_nomatch)

cond2_match = (
    ~np.isnan(pwlaux_nomatch['lr']) &
    (pwlaux_nomatch['lr'] >= threshold_sel)
)
m2_m = np.sum(cond2_match)
print(m2_m)

cond2_nomatch = (
    np.isnan(pwlaux_nomatch['lr']) |
    (pwlaux_nomatch['lr'] < threshold_sel)
)
m2_nm = np.sum(cond2_nomatch)
print(m2_nm)

m2_nm + m2_m

m2_nm + m2_m + m_nm + m_dm + m_m


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Plots for the paper
q(m) over n(m) depending on the colour (not used)
Can you write Python code for it?

plt.rcParams["figure.figsize"] = (12,10)

from matplotlib import cm
from matplotlib.collections import LineCollection

cm_subsection = linspace(0., 1., 16)
colors = [ cm.viridis(x) for x in cm_subsection ]

low = np.nonzero(centers[1] >= 15)[0][0]
high = np.nonzero(centers[1] >= 22.2)[0][0]

fig, a = plt.subplots()

for i, q_m_k in enumerate(q_m):
    #plot(centers[i], q_m_old[i]/n_m_old[i])
    a = subplot(4,4,i+1)
    if i not in [-1]:
        q_m_aux = q_m[i]/np.sum(q_m[i])
        lwidths = (q_m_aux/np.max(q_m_aux)*10).astype(float)
        #print(lwidths)

        y_aux = q_m_k/n_m[i]
        factor = np.max(y_aux[low:high])
        y = y_aux
        #print(y)
        x = centers[i]

        points = np.array([x, y]).T.reshape(-1, 1, 2)
        segments = np.concatenate([points[:-1], points[1:]], axis=1)
        lc = LineCollection(segments, linewidths=lwidths, color=colors[i])

        a.add_collection(lc)

        #plot(centers[i], x/factor, color=colors[i-1])
        xlim([12, 30])
        if i == 0:
            xlim([10, 23])
        ylim([0, 1.2*factor])

subplots_adjust(left=0.125,
                bottom=0.1,
                right=0.9,
                top=0.9,
                wspace=0.4,
                hspace=0.2)


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Save data for tests
Can you write Python code for it?

np.sum(~np.isnan(pwl["colour"]) & (pwl["match_code"] == 3))

subplot(1,2,1)
val, bins, _ = hist(pwl["colour"][~np.isnan(pwl["colour"]) & (pwl["match_code"] != 0)],
                    bins=50, normed=True, alpha=0.5, label="Total")
val, bins, _ = hist(pwl["colour"][~np.isnan(pwl["colour"]) & (pwl["match_code"] == 4)],
                    bins=bins, normed=True, alpha=0.5, label="New matches")
xlabel("Colour")
ylabel("N normed")
legend()
subplot(1,2,2)
val, bins, _ = hist(pwl["colour"][~np.isnan(pwl["colour"]) & (pwl["match_code"] != 0)],
                    bins=bins, normed=True, alpha=0.5, label="Total")
val, bins, _ = hist(pwl["colour"][~np.isnan(pwl["colour"]) & (pwl["match_code"] == 2)],
                    bins=bins, normed=True, alpha=0.5, label="Different match")
xlabel("Colour")
ylabel("N normed")
legend()


I am working on the file "Match_LOFAR_analysis_final.ipynb".
The first task is:
Study the 3 repeated sources
3 sources that are in group 1 and 2
Can you write Python code for it?

pwl['match_code2'] = 0
pwl['match_code2'][np.isin(pwl["Source_Name"], pwl[cond_mlr & ~np.isnan(pwl['ML_LR'])][cond_match]["Source_Name"])]=1

t = pwl[np.isin(pwl["Source_Name"], pwl[cond_mlr & ~np.isnan(pwl['ML_LR'])][cond_diffmatch]["Source_Name"])]
t[t['match_code2'] != 0][['Source_Name', "AllWISE", "AllWISE_input", "objID", "objID_input"]]

pwl[pwl["objID"] == 164861629587942860]


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Additional description of the data
Can you write Python code for it?


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Analyse changes in the matches
Can you write Python code for it?


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Change the AllWISE_input that are '' to 'N/A'. That comes from a previous error with the fill value.
Can you write Python code for it?


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Create the additional columns for the types of matches
Can you write Python code for it?

threshold_sel = 0.639

cond_mlr = (pwl['ID_flag'] == 1) & (pwl['Maj'] < 30.)

pwlaux = pwl[cond_mlr].filled()


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Explanation of the KDE used in the computing of the q(m) and n(m):
* https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/
Figure 3 of the paper
Help from:
* https://stackoverflow.com/questions/19877666/add-legends-to-linecollection-plot
* https://matplotlib.org/users/mathtext.html#mathtext-tutorial
Can you write Python code for it?

save = True
save_pdf = True

plt.rcParams["figure.figsize"] = (6.64, 6.64*0.74)
plt.rcParams["figure.dpi"] = 100
plt.rcParams['lines.linewidth'] = 1.75
plt.rcParams['lines.markersize'] = 8.0
plt.rcParams['lines.markeredgewidth'] = 0.75
plt.rcParams['font.size'] = 15.0 ## not 18.0
plt.rcParams['font.family'] = "serif"
plt.rcParams['font.serif'] = "CM"
plt.rcParams['xtick.labelsize'] = 'small'
plt.rcParams['ytick.labelsize'] = 'small'
plt.rcParams['xtick.major.width'] = 1.0
plt.rcParams['xtick.major.size'] = 8
plt.rcParams['ytick.major.width'] = 1.0
plt.rcParams['ytick.major.size'] = 8
plt.rcParams['xtick.minor.width'] = 1.0
plt.rcParams['xtick.minor.size'] = 4
plt.rcParams['ytick.minor.width'] = 1.0
plt.rcParams['ytick.minor.size'] = 4
plt.rcParams['axes.linewidth'] = 1.5
plt.rcParams['legend.fontsize'] = 'small'
plt.rcParams['legend.numpoints'] = 1
plt.rcParams['legend.labelspacing'] = 0.4
plt.rcParams['legend.frameon'] = False
plt.rcParams['text.usetex'] = True
plt.rcParams['savefig.dpi'] = 300

from matplotlib import cm
from matplotlib.collections import LineCollection
from matplotlib.lines import Line2D

cm_subsection = linspace(0., 1., 16)
colors = [ cm.viridis(x) for x in cm_subsection ]

low = np.nonzero(centers[1] >= 15)[0][0]
high = np.nonzero(centers[1] >= 22.2)[0][0]

cm_subsection = linspace(0., 1., 14)
colors = [ cm.jet(x) for x in cm_subsection ]

fig, a = plt.subplots()
lcs = []
proxies = []

def make_proxy(zvalue, scalar_mappable, **kwargs):
    color = scalar_mappable.cmap(scalar_mappable.norm(zvalue))
    return Line2D([0, 1], [0, 1], color=color, **kwargs)

for i, q_m_k in enumerate(q_m):
    #plot(centers[i], q_m_old[i]/n_m_old[i])
    q_m_aux = q_m[i]/np.sum(q_m[i])
    lwidths = (q_m_aux/np.max(q_m_aux)*10).astype(float)
    if save_pdf and (i<6): # Solve problems with the line with in pdfs
        lwidths[lwidths < 0.005] = 0
    #print(lwidths)

    y_aux = q_m_k/n_m[i]
    factor = np.max(y_aux[low:high])
    y = y_aux
    #print(y)
    x = centers[i]

    points = np.array([x,...


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Explore and repair data
Can you write Python code for it?


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Figure 4 of the paper: completeness and reliability
Can you write Python code for it?

q0 = np.sum(Q_0_colour)

def completeness(lr, threshold, q0):
    n = len(lr)
    lrt = lr[lr < threshold]
    return 1. - np.sum((q0 * lrt)/(q0 * lrt + (1 - q0)))/float(n)/q0

def reliability(lr, threshold, q0):
    n = len(lr)
    lrt = lr[lr > threshold]
    return 1. - np.sum((1. - q0)/(q0 * lrt + (1 - q0)))/float(n)/q0

completeness_v = np.vectorize(completeness, excluded=[0])
reliability_v = np.vectorize(reliability, excluded=[0])

pwl["lrt"] = pwl["lr"]
pwl["lrt"][np.isnan(pwl["lr"])] = 0

n_test = 100
threshold_mean = np.percentile(pwl["lrt"], 100*(1 - q0))

thresholds = np.arange(0., 10., 0.01)
thresholds_fine = np.arange(0.1, 2., 0.001)
completeness_t = completeness_v(pwl["lrt"], thresholds, q0)
reliability_t = reliability_v(pwl["lrt"], thresholds, q0)
average_t = (completeness_t + reliability_t)/2

completeness_t_fine = completeness_v(pwl["lrt"], thresholds_fine, q0)
reliability_t_fine = reliability_v(pwl["lrt"], thresholds_fine, q0)
average_t_fine = (completeness_t_fine + reliability_t_fine)/2

thresholds_fine[np.argmax(average_t_fine)]

thresholds_fine[np.argmin(np.abs(completeness_t_fine-reliability_t_fine))]

np.sum(pwl["lrt"] >= 0.358)

np.sum(pwl["lrt"] >= 0.639)

np.sum((pwl["lrt"] >= 0.639) & (pwl['ID_flag'] == 1) & ~np.isnan(pwl['ra']))

len(pwl)

len(pwl[pwl['ID_flag'] == 1])

n0 = np.sum((pwl["lrt"] >= 0.639) & (pwl['ID_flag'] == 1))
print(n0, n0/len(pwl[pwl['ID_flag'] == 1]))

n0 = np.sum((pwl["lrt"] >= 0.81268) & (pwl['ID_flag'] == 1))
print(n0, n0/len(pwl[pwl['ID_flag'] == 1]))

n0 = np.sum((pwl["ML_LR"] >= 0.81268) & (pwl['ID_flag'] == 1))
print(n0, n0/len(pwl[pwl['ID_flag'] == 1]))

np.sum(pwl["lrt"] >= 0.639)/len(pwl)

threshold_sel = 0.639

thresholds_fine = np.arange(0.0, 2., 0.001)
completeness_t_fine = completeness_v(pwl["lrt"], thresholds_fine, q0)
reliability_t_fine = reliability_v(pwl["lrt"], thresholds_fine, q0)

plt.rcParams["figure.figsize"] = (6.64, 6.64*0.74)
plt.rcParams["figure.dpi"] = 100
plt.rcParams['lines.linewidth'] = 1.75
plt.rcParams['lines.markersize'] =...


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Fractions
Load the full catalogue of galaxies and work out the fraction of matched LOFAR. It will be necessary to work in the restricted area with both the pw and the lofar matched catalogue.
The fractions could be corrected if the relative fractions of sources of each type change between the full area and the restricted area for the LOFAR matches.
Can you write Python code for it?

field = Field(170.0, 190.0, 46.8, 55.9)

combined = Table.read("pw.fits")

combined = field.filter_catalogue(combined,
                               colnames=("ra", "dec"))

combined["colour"] = combined["i"] - combined["W1mag"]

colour_limits = [0.0, 0.5, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 2.75, 3.0, 3.5, 4.0]

combined_panstarrs = (~np.isnan(combined["i"]) & np.isnan(combined["W1mag"])) # Sources with only i-band
combined_wise =(np.isnan(combined["i"]) & ~np.isnan(combined["W1mag"])) # Sources with only W1-band

# Start with the W1-only, i-only and "less than lower colour" bins
colour_bin_def = [{"name":"only W1", "condition": combined_wise},
                  {"name":"only i", "condition": combined_panstarrs},
                  {"name":"-inf to {}".format(colour_limits[0]),
                   "condition": (combined["colour"] < colour_limits[0])}]

# Get the colour bins
for i in range(len(colour_limits)-1):
    name = "{} to {}".format(colour_limits[i], colour_limits[i+1])
    condition = ((combined["colour"] >= colour_limits[i]) &
                 (combined["colour"] < colour_limits[i+1]))
    colour_bin_def.append({"name":name, "condition":condition})

# Add the "more than higher colour" bin
colour_bin_def.append({"name":"{} to inf".format(colour_limits[-1]),
                       "condition": (combined["colour"] >= colour_limits[-1])})

combined["category"] = np.nan
for i in range(len(colour_bin_def)):
    combined["category"][colour_bin_def[i]["condition"]] = i

numbers_combined_bins = np.array([np.sum(a["condition"]) for a in colour_bin_def])

numbers_combined_bins

pwlf = field.filter_catalogue(pwl[cond_mlr],
                               colnames=("ra", "dec"))

pwlf_all = field.filter_catalogue(pwl,
                               colnames=("ra", "dec"))

lofar_c, pw_c, lofar_c_all = [], [], []
avg_colour = []
for i in range(16):
    lofar_c.append(np.sum(pwlf["category"] == i))
    lofar_c_all.append(np.sum(pwlf_all["category"] == i))
    pw_c.append(np.sum(combined["category"] == i))
  ...


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
General configuration
Can you write Python code for it?

save_intermediate = True
plot_intermediate = True

idp = "idata/final_analysis_pdf_v2.0"

if not os.path.isdir(idp):
    os.makedirs(idp)


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Load data
Can you write Python code for it?

pwli = Table.read("lofar_pw_pdf.fits")

pwli.colnames

lofar_all = Table.read("data/LOFAR_HBA_T1_DR1_merge_ID_optical_f_v1.2.fits")


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Matched sources
Can you write Python code for it?


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
N_LOFAR
Can you write Python code for it?

cond_mlr = (pwl['ID_flag'] == 1) & (pwl["lrt"] >= 0.639)

len(pwl)

np.sum(np.isnan(pwl["category"]))

np.sum(np.isnan(pwl["category"][cond_mlr]))

n_c, n_c_mlr = [], []
for i in np.unique(pwl["category"][~np.isnan(pwl["category"])]):
    n_c.append(np.sum((pwl["category"] == i)))
    n_c_mlr.append(np.sum((pwl["category"][cond_mlr] == i)))
    print(i, n_c[-1], n_c_mlr[-1])

total_n_c = np.sum(n_c)
total_n_c_mlr = np.sum(n_c_mlr)
print(total_n_c, total_n_c_mlr)
print(len(pwl)-np.sum(np.isnan(pwl["category"])), len(pwl)-np.sum(np.isnan(pwl["category"][cond_mlr])))
for i in range(16):
    print("{:2d} {:6d} {:6d} {:6.3f} {:6.3f} {:.2%}".format(i,
                                                     n_c[i],
                                                     n_c_mlr[i],
                                                     n_c[i]/total_n_c*100,
                                                     n_c_mlr[i]/total_n_c_mlr*100,
                                                     (n_c[i]-n_c_mlr[i])/n_c[i]
         ))

for i in range(16):
    print("{:2d} {:6d}".format(i, n_c_mlr[i] ))


I am working on the file "Match_LOFAR_analysis_final_numbers.ipynb".
The first task is:
Non-matched sources
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_above-threshold.ipynb".
The first task is:
Additional data
Can you write Python code for it?

combined["colour"] = combined["i"] - combined["W1mag"]

combined_aux_index = np.arange(len(combined))


I am working on the file "Match_LOFAR_combined_above-threshold.ipynb".
The first task is:
Area limits
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_above-threshold.ipynb".
The first task is:
Class of sources in the combined catalogue
The sources are grouped depending on the available photometric data.
Can you write Python code for it?

combined_matched = (~np.isnan(combined["i"]) & ~np.isnan(combined["W1mag"])) # Matched i-W1 sources

combined_panstarrs = (~np.isnan(combined["i"]) & np.isnan(combined["W1mag"])) # Sources with only i-band

combined_wise =(np.isnan(combined["i"]) & ~np.isnan(combined["W1mag"])) # Sources with only W1-band

combined_i = combined_matched | combined_panstarrs
combined_w1 = combined_matched | combined_wise
#combined_only_i = combined_panstarrs & ~combined_matched
#combined_only_w1 = combined_wise & ~combined_matched

print("Total    - ", len(combined))
print("i and W1 - ", np.sum(combined_matched))
print("Only i   - ", np.sum(combined_panstarrs))
print("With i   - ", np.sum(combined_i))
print("Only W1  - ", np.sum(combined_wise))
print("With W1  - ", np.sum(combined_w1))


I am working on the file "Match_LOFAR_combined_above-threshold.ipynb".
The first task is:
Colour categories
The colour categories will be used after the first ML match
Can you write Python code for it?

colour_limits = [-0.5, 0.0, 0.5, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 2.75, 3.0, 3.5, 4.0]

# Start with the W1-only, i-only and "less than lower colour" bins
colour_bin_def = [{"name":"only W1", "condition": combined_wise},
                  {"name":"only i", "condition": combined_panstarrs},
                  {"name":"-inf to {}".format(colour_limits[0]),
                   "condition": (combined["colour"] < colour_limits[0])}]

# Get the colour bins
for i in range(len(colour_limits)-1):
    name = "{} to {}".format(colour_limits[i], colour_limits[i+1])
    condition = ((combined["colour"] >= colour_limits[i]) &
                 (combined["colour"] < colour_limits[i+1]))
    colour_bin_def.append({"name":name, "condition":condition})

# Add the "more than higher colour" bin
colour_bin_def.append({"name":"{} to inf".format(colour_limits[-1]),
                       "condition": (combined["colour"] >= colour_limits[-1])})

combined["category"] = np.nan
for i in range(len(colour_bin_def)):
    combined["category"][colour_bin_def[i]["condition"]] = i

np.sum(np.isnan(combined["category"]))


I am working on the file "Match_LOFAR_combined_above-threshold.ipynb".
The first task is:
Load data
Can you write Python code for it?

combined_all = Table.read("pw.fits")

lofar_all = Table.read("data/LOFAR_HBA_T1_DR1_catalog_v0.9.srl.fits")

np.array(combined_all.colnames)

np.array(lofar_all.colnames)


I am working on the file "Match_LOFAR_combined_above-threshold.ipynb".
The first task is:
ML match
Can you write Python code for it?

radius = 15

lr_threshold = 0.36

selection = ~np.isnan(combined["category"]) # Avoid the dreaded sources with no actual data
catalogue = combined[selection]

def apply_ml(i, likelihood_ratio_function):
    idx_0 = idx_i[idx_lofar == i]
    d2d_0 = d2d[idx_lofar == i]

    category = catalogue["category"][idx_0].astype(int)
    mag = catalogue["i"][idx_0]
    mag[category == 0] = catalogue["W1mag"][idx_0][category == 0]

    lofar_ra = lofar[i]["RA"]
    lofar_dec = lofar[i]["DEC"]
    lofar_pa = lofar[i]["PA"]
    lofar_maj_err = lofar[i]["E_Maj"]
    lofar_min_err = lofar[i]["E_Min"]
    c_ra = catalogue["ra"][idx_0]
    c_dec = catalogue["dec"][idx_0]
    c_ra_err = catalogue["raErr"][idx_0]
    c_dec_err = catalogue["decErr"][idx_0]

    sigma, sigma_maj, sigma_min = get_sigma_all(lofar_maj_err, lofar_min_err, lofar_pa,
                      lofar_ra, lofar_dec,
                      c_ra, c_dec, c_ra_err, c_dec_err)

    lr_0 = likelihood_ratio_function(mag, d2d_0.arcsec, sigma, sigma_maj, sigma_min, category)

    chosen_index = (lr_0 >= lr_threshold)
    result = [combined_aux_index[selection][idx_0[chosen_index]], # Index
              (d2d_0.arcsec)[chosen_index],                        # distance
              lr_0[chosen_index]]                                  # LR
    return result


I am working on the file "Match_LOFAR_combined_above-threshold.ipynb".
The first task is:
Run the ML matching
Can you write Python code for it?

def ml(i):
    return apply_ml(i, likelihood_ratio_function)

res = parallel_process(idx_lofar_unique, ml, n_jobs=8)


I am working on the file "Match_LOFAR_combined_above-threshold.ipynb".
The first task is:
We get the number of sources of the combined catalogue in each colour category. It will be used at a later stage to compute the $Q_0$ values
Can you write Python code for it?

numbers_combined_bins = np.array([np.sum(a["condition"]) for a in colour_bin_def])

numbers_combined_bins


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
Additional data
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
Area limits
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
Class of sources in the combined catalogue
The sources are grouped depending on the available photometric data.
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
Colour categories
The colour categories will be used after the first ML match
Can you write Python code for it?

colour_limits = [0.0, 0.5, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 2.75, 3.0, 3.5, 4.0]

# Start with the W1-only, i-only and "less than lower colour" bins
colour_bin_def = [{"name":"only W1", "condition": combined_wise},
                  {"name":"only i", "condition": combined_panstarrs},
                  {"name":"-inf to {}".format(colour_limits[0]),
                   "condition": (combined["colour"] < colour_limits[0])}]

# Get the colour bins
for i in range(len(colour_limits)-1):
    name = "{} to {}".format(colour_limits[i], colour_limits[i+1])
    condition = ((combined["colour"] >= colour_limits[i]) &
                 (combined["colour"] < colour_limits[i+1]))
    colour_bin_def.append({"name":name, "condition":condition})

# Add the "more than higher colour" bin
colour_bin_def.append({"name":"{} to inf".format(colour_limits[-1]),
                       "condition": (combined["colour"] >= colour_limits[-1])})

combined["category"] = np.nan
for i in range(len(colour_bin_def)):
    combined["category"][colour_bin_def[i]["condition"]] = i

np.sum(np.isnan(combined["category"]))


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
Load data
Can you write Python code for it?

combined_all = Table.read("pw.fits")

lofar_all = Table.read("data/LOFAR_HBA_T1_DR1_catalog_v0.9.srl.fixed.fits")
#lofar_all = Table.read("data/LOFAR_HBA_T1_DR1_merge_ID_optical_v0.8.fits")

np.array(combined_all.colnames)

np.array(lofar_all.colnames)


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
ML match
Can you write Python code for it?

radius = 15

selection = ~np.isnan(combined["category"]) # Avoid the dreaded sources with no actual data
catalogue = combined[selection]

def apply_ml(i, likelihood_ratio_function):
    idx_0 = idx_i[idx_lofar == i]
    d2d_0 = d2d[idx_lofar == i]

    category = catalogue["category"][idx_0].astype(int)
    mag = catalogue["i"][idx_0]
    mag[category == 0] = catalogue["W1mag"][idx_0][category == 0]

    lofar_ra = lofar[i]["RA"]
    lofar_dec = lofar[i]["DEC"]
    lofar_pa = lofar[i]["PA"]
    lofar_maj_err = lofar[i]["E_Maj"]
    lofar_min_err = lofar[i]["E_Min"]
    c_ra = catalogue["ra"][idx_0]
    c_dec = catalogue["dec"][idx_0]
    c_ra_err = catalogue["raErr"][idx_0]
    c_dec_err = catalogue["decErr"][idx_0]

    sigma_0_0, det_sigma = get_sigma_all(lofar_maj_err, lofar_min_err, lofar_pa,
                      lofar_ra, lofar_dec,
                      c_ra, c_dec, c_ra_err, c_dec_err)

    lr_0 = likelihood_ratio_function(mag, d2d_0.arcsec, sigma_0_0, det_sigma, category)

    chosen_index = np.argmax(lr_0)
    result = [combined_aux_index[selection][idx_0[chosen_index]], # Index
              (d2d_0.arcsec)[chosen_index],                        # distance
              lr_0[chosen_index]]                                  # LR
    return result


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
Maximum Likelihood
Can you write Python code for it?

bin_list, centers, Q_0_colour, n_m, q_m = pickle.load(open("lofar_params.pckl", "rb"))

likelihood_ratio_function = MultiMLEstimator(Q_0_colour, n_m, q_m, centers)


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
Run the ML matching
Can you write Python code for it?

import multiprocessing
n_cpus_total = multiprocessing.cpu_count()
n_cpus = max(1, n_cpus_total-1)

def ml(i):
    return apply_ml(i, likelihood_ratio_function)

res = parallel_process(idx_lofar_unique, ml, n_jobs=n_cpus)

lofar["lr"] = np.nan                   # Likelihood ratio
lofar["lr_dist"] = np.nan              # Distance to the selected source
lofar["lr_index"] = np.nan             # Index of the PanSTARRS source in combined

(lofar["lr_index"][idx_lofar_unique],
 lofar["lr_dist"][idx_lofar_unique],
 lofar["lr"][idx_lofar_unique]) = list(map(list, zip(*res)))

total_sources = len(idx_lofar_unique)
combined_aux_index = np.arange(len(combined))


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
Run the cross-match
Can you write Python code for it?

idx_lofar, idx_i, d2d, d3d = search_around_sky(
    coords_lofar, coords_combined[selection], radius*u.arcsec)

idx_lofar_unique = np.unique(idx_lofar)


I am working on the file "Match_LOFAR_combined_final.ipynb".
The first task is:
Threshold and selection
Can you write Python code for it?

lofar["lrt"] = lofar["lr"]
lofar["lrt"][np.isnan(lofar["lr"])] = 0

q0 = np.sum(Q_0_colour)

def completeness(lr, threshold, q0):
    n = len(lr)
    lrt = lr[lr < threshold]
    return 1. - np.sum((q0 * lrt)/(q0 * lrt + (1 - q0)))/float(n)/q0

def reliability(lr, threshold, q0):
    n = len(lr)
    lrt = lr[lr > threshold]
    return 1. - np.sum((1. - q0)/(q0 * lrt + (1 - q0)))/float(n)/q0

completeness_v = np.vectorize(completeness, excluded=[0])
reliability_v = np.vectorize(reliability, excluded=[0])

n_test = 100
threshold_mean = np.percentile(lofar["lrt"], 100*(1 - q0))

thresholds = np.arange(0., 10., 0.01)
thresholds_fine = np.arange(0.1, 1., 0.001)

completeness_t = completeness_v(lofar["lrt"], thresholds, q0)
reliability_t = reliability_v(lofar["lrt"], thresholds, q0)
average_t = (completeness_t + reliability_t)/2

completeness_t_fine = completeness_v(lofar["lrt"], thresholds_fine, q0)
reliability_t_fine = reliability_v(lofar["lrt"], thresholds_fine, q0)
average_t_fine = (completeness_t_fine + reliability_t_fine)/2

# Maximum of the average
thresholds_fine[np.argmax(average_t_fine)]

# Crossing point of completeness and reliability
thresholds_fine[np.argmin(np.abs(completeness_t_fine-reliability_t_fine))]

# Value selected for the paper
threshold_sel = 0.639

threshold_sel = 0.601

plt.rcParams["figure.figsize"] = (15,6)
subplot(1,2,1)
plot(thresholds, completeness_t, "r-")
plot(thresholds, reliability_t, "g-")
plot(thresholds, average_t, "k-")
vlines(threshold_sel, 0.9, 1., "k", linestyles="dashed")
vlines(threshold_mean, 0.9, 1., "y", linestyles="dashed")
ylim([0.9, 1.])
xlabel("Threshold")
ylabel("Completeness/Reliability")
subplot(1,2,2)
plot(thresholds_fine, completeness_t_fine, "r-")
plot(thresholds_fine, reliability_t_fine, "g-")
plot(thresholds_fine, average_t_fine, "k-")
vlines(threshold_sel, 0.9, 1., "k", linestyles="dashed")
#vlines(threshold_mean, 0.9, 1., "y", linestyles="dashed")
ylim([0.97,...


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
Additional data
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
Area limits
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
Class of sources in the combined catalogue
The sources are grouped depending on the available photometric data.
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
Colour categories
The colour categories will be used after the first ML match
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
Filter catalogues
The following line has been corrected in the latest versions to use all the sources, including the extended. Hence the running of the "-extended" version of this notebook is no longer necessary.
Can you write Python code for it?

lofar = field_full.filter_catalogue(lofar_all, colnames=("RA", "DEC"))

combined = field_full.filter_catalogue(combined_all,
                               colnames=("ra", "dec"))


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
ML match
Can you write Python code for it?

radius = 15

selection = ~np.isnan(combined["category"]) # Avoid the dreaded sources with no actual data
catalogue = combined[selection]

def apply_ml(i, likelihood_ratio_function):
    idx_0 = idx_i[idx_lofar == i]
    d2d_0 = d2d[idx_lofar == i]

    category = catalogue["category"][idx_0].astype(int)
    mag = catalogue["i"][idx_0]
    mag[category == 0] = catalogue["W1mag"][idx_0][category == 0]

    lofar_ra = lofar[i]["RA"]
    lofar_dec = lofar[i]["DEC"]
    lofar_pa = lofar[i]["PA"]
    lofar_maj_err = lofar[i]["E_Maj"]
    lofar_min_err = lofar[i]["E_Min"]
    c_ra = catalogue["ra"][idx_0]
    c_dec = catalogue["dec"][idx_0]
    c_ra_err = catalogue["raErr"][idx_0]
    c_dec_err = catalogue["decErr"][idx_0]

    sigma, sigma_maj, sigma_min = get_sigma_all(lofar_maj_err, lofar_min_err, lofar_pa,
                      lofar_ra, lofar_dec,
                      c_ra, c_dec, c_ra_err, c_dec_err)

    lr_0 = likelihood_ratio_function(mag, d2d_0.arcsec, sigma, sigma_maj, sigma_min, category)

    chosen_index = np.argmax(lr_0)
    result = [combined_aux_index[selection][idx_0[chosen_index]], # Index
              (d2d_0.arcsec)[chosen_index],                        # distance
              lr_0[chosen_index]]                                  # LR
    return result


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
Run the ML matching
Can you write Python code for it?

def ml(i):
    return apply_ml(i, likelihood_ratio_function)

res = parallel_process(idx_lofar_unique, ml, n_jobs=8)

lofar["lr"] = np.nan                   # Likelihood ratio
lofar["lr_dist"] = np.nan              # Distance to the selected source
lofar["lr_index"] = np.nan             # Index of the PanSTARRS source in combined

(lofar["lr_index"][idx_lofar_unique],
 lofar["lr_dist"][idx_lofar_unique],
 lofar["lr"][idx_lofar_unique]) = list(map(list, zip(*res)))

total_sources = len(idx_lofar_unique)
combined_aux_index = np.arange(len(combined))


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
Run the cross-match
Can you write Python code for it?


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
Sky coordinates
Can you write Python code for it?

coords_combined = SkyCoord(combined['ra'],
                           combined['dec'],
                           unit=(u.deg, u.deg),
                           frame='icrs')

coords_lofar = SkyCoord(lofar['RA'],
                       lofar['DEC'],
                       unit=(u.deg, u.deg),
                       frame='icrs')


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
Threshold and selection
Can you write Python code for it?

lofar["lrt"] = lofar["lr"]
lofar["lrt"][np.isnan(lofar["lr"])] = 0

threshold_sel = 0.358

lofar["lr_index_sel"] = lofar["lr_index"]
lofar["lr_index_sel"][lofar["lrt"] < threshold_sel] = np.nan


I am working on the file "Match_LOFAR_combined_generic.ipynb".
The first task is:
We get the number of sources of the combined catalogue in each colour category. It will be used at a later stage to compute the $Q_0$ values
Can you write Python code for it?


I am working on the file "Matching Market v9.ipynb".
The first task is:
Example Market
In the following code example we use the buyer and supplier objects to create a market. At the market a single price is announced which causes as many units of goods to be swapped as possible. The buyers and sellers stop trading when it is no longer in their own interest to continue.
Can you write Python code for it?

# Show some real consumption data, for more data see folder data analytics
#read montly consumption data of 2010 into a dataframe
df = pd.read_csv('2010cbstestrun.csv', header=0, index_col=0)
df = df.transpose()

#plot the 2010 monthly consumption data
df.plot();
df

# make initialization dictionary
init_buyer = {'elec':{'b':400, 'm' : 673, 'lb_price': 10, 'ub_price' : 20},
              'indu':{'b':400, 'm':1171, 'lb_price': 10, 'ub_price' : 20},
              'home':{'b': 603, 'm': 3615, 'lb_price': 10, 'ub_price' : 20}}
init_seller = {'NL' : {'prod': 2000, 'lb_price': 10, 'ub_price' : 20, 'reserve': 50000},
               'RU' : {'prod': 2000, 'lb_price': 15, 'ub_price' : 30, 'reserve': 500000}}

# make a history book to record every timestep
hist_book = []

# set the starting time
period_null= datetime.date(2010,1,1)


I am working on the file "Matching Market v9.ipynb".
The first task is:
Observer
The observer holds the clock and collects data. In this setup it tells the market another tick has past and it is time to act. The market will instruct the other agents. The observer initializes the model, thereby making real objects out of the classes defined above.
Can you write Python code for it?

class Observer():
    def __init__(self, x, y, z):
        self.init_buyer = x
        self.init_seller = y
        self.maxrun = z
        self.hist_book = []
        self.buyer_dict = {}
        self.seller_dict = {}
        self.timetick = 0
        self.gas_market = ''
        self.reserve = []

    def set_buyer(self, buyer_info):
        for name in buyer_info:
            self.buyer_dict[name] = Buyer('%s' % name)
            self.buyer_dict[name].base_demand = buyer_info[name]['b']
            self.buyer_dict[name].max_demand = buyer_info[name]['m']
            self.buyer_dict[name].lb_price = buyer_info[name]['lb_price']
            self.buyer_dict[name].ub_price = buyer_info[name]['ub_price']

    def set_seller(self, seller_info):
        for name in seller_info:
            self.seller_dict[name] = Seller('%s' % name)
            self.seller_dict[name].prod = seller_info[name]['prod']
            self.seller_dict[name].lb_price = seller_info[name]['lb_price']
            self.seller_dict[name].ub_price = seller_info[name]['ub_price']
            self.seller_dict[name].reserve = seller_info[name]['reserve']
            self.seller_dict[name].init_reserve = seller_info[name]['reserve']

    def get_reserve(self):
        reserve = []
        for name in self.seller_dict:
            reserve.append(self.seller_dict[name].reserve)
        return reserve

    def set_market(self):
        self.gas_market = Market()
        #add suplliers and buyers to this market
        for supplier in self.seller_dict.values():
            self.gas_market.add_seller(supplier)
        for buyer in self.buyer_dict.values():
            self.gas_market.add_buyer(buyer)
        self.gas_market.seller_dict = self.seller_dict
        self.gas_market.buyer_dict = self.buyer_dict

    def run_it(self):
        # Timing
        # time initialising
        startit_init = time.time()

        #initialise, setting up all the agents
        first_run = True
        if first_run:
           ...


I am working on the file "Matching Market v9.ipynb".
The first task is:
Time of last run
Time and date of the last run of this notebook file
Can you write Python code for it?

# print the time of last run
print('last run of this notebook:')
time.strftime("%a, %d %b %Y %H:%M:%S", time.localtime())

#df_hb
df_res = pd.DataFrame(obser1.reserve, columns=['time', *[i for i in init_seller]])
df_res = df_res.set_index('time')
df_res.plot();

df_res['NL']


I am working on the file "Matching Market v9.ipynb".
The first task is:
classes buyers and sellers
Below we are constructing the buyers and sellers in classes.
Can you write Python code for it?

# measure how long it takes to run the script
startit = time.time()
dtstartit = datetime.datetime.now()

class Seller():
    def __init__(self, name):
        self.name = name
        self.wta = []
        self.step = 0
        self.prod = 2000
        self.lb_price = 10
        self.lb_multiplier = 0
        self.ub_price = 20
        self.ub_multiplier = 0
        self.init_reserve = 500000
        self.reserve = 500000
        #multiple market idea, also 'go away from market'
        self.subscr_market = {}

    # the supplier has n quantities that they can sell
    # they may be willing to sell this quantity anywhere from a lower price of l
    # to a higher price of u
    def set_quantity(self):
        self.update_price()
        n = self.prod
        l = self.lb_price + self.lb_multiplier
        u = self.ub_price + self.ub_multiplier
        wta = []
        for i in range(n):
            p = rnd.uniform(l, u)
            wta.append(p)

        if len(wta) < self.reserve:
            self.wta = wta
        else:
            self.wta = wta[0:(self.reserve-1)]
            self.prod = self.reserve

    def get_name(self):
        return self.name

    def get_asks(self):
        return self.wta

    def extract(self, cur_extraction):
        if self.reserve > 0:
            self.reserve = self.reserve - cur_extraction
        else:
            self.prod = 0

    # production costs rise a 100%
    def update_price(self):
        depletion = (self.init_reserve - self.reserve) / self.init_reserve
        self.ub_multiplier = int(self.ub_price * depletion)
        self.lb_multiplier = int(self.lb_price * depletion)

class Buyer():
    def __init__(self, name):
        self.name = name
        self.type = 0
        self.wtp = []
        self.step = 0
        self.base_demand = 0
        self.max_demand = 0
        self.lb_price = 10
        self.ub_price = 20

    # the supplier has n quantities that they can buy
    # they may be willing to sell this quantity anywhere from a lower price of...


I am working on the file "Matching Market v9.ipynb".
The first task is:
run the model
To run the model we create the observer. The observer creates all the other objects and runs the model.
Can you write Python code for it?

# create observer and run the model
# first data about buyers then sellers and then model ticks
years = 10
timestep = 12

obser1 = Observer(init_buyer, init_seller, years*timestep)
obser1.run_it()
#get the info from the observer
hist_book = obser1.hist_book

# recording the total run
def write_to_csv(hist_book):
    f = open('hist_book.csv', 'a')
    for item in hist_book:
        f.write('%s,%s\n' % (item[0], item[1]))
    f.close()

#write_to_csv(hist_book)

# make a dataframe of clearing prices
df_hb = pd.DataFrame(hist_book)
df_hb = df_hb.set_index(0)
df_hb.index.name = 'month'
df_hb.rename(columns={1: 'price', 2: 'quantity'}, inplace=True)


I am working on the file "MathConceptsForDevelopers-Sep17/04_Hight-SchoolMaths-E/High-School Maths Exercise/Solutions.ipynb".
The first task is:
High-School Maths Exercise - Solutions
Basic techniques and concepts for working with Jupyter Notebook, Python and Python libraries

Hristo Kostov

github link to this solution

Solution 1. Markdown
lets try some sorted numbered list

this is the first point
this is inclined point
this is sedonc inclined point
This is second main point

Now lets try unsorted list

this is a point
and this is inclined point
second indlined point
Again some point
this will have sublist
this is sublist
of few elements
handing here

One table
| SoftUni | Google |
|------------------------------|---------------------------:|
| 1. |  and if you click this  |
| 2. | * it will take you * |
| 3. | to SoftUni or Google |
Solution 2. Quadratic Equations - formulas and LaTex
We start with the equation
$$ ax^2 + bx + c = 0 $$
We can try to get something like the sum of squares formula. Recall that
$$ (px+q)^2 = p^2x^2 + 2pqx + q^2 $$
Let's take the first two terms ($ax^2 + bx$). We can see that they fit the formula above almost perfectly. We have
$$ a = p^2, b = 2pq $$
$$ \Rightarrow p = \sqrt{a}, q = \frac{b}{2p} = \frac{b}{2\sqrt{a}}$$.
We also need to add $q^2=\frac{b^2}{4a}$. Since this is an equation, we have to add it to both sides of the original equation. We get:
$$ p^2x^2 + 2pqx + q^2 + c = q^2$$
Now, express the equation above in terms of $a, b, c$:

Result:

$$ {\sqrt{a}}^2x^2 + 2\sqrt{a}\frac{b}{2\sqrt{a}}x + \frac{b^2}{4a} + c = \frac{b^2}{4a} $$
Place $c$ on the right-hand side. We now have our squared formula on the left:

Result:

$$ \sqrt{a}^2x^2 + 2\sqrt{a}\frac{b}{2\sqrt{a}}x + \frac{b^2}{4a} = \frac{b^2}{4a} - c $$
$$ \left(\sqrt{a}\right)^2x^2 + 2\left(\sqrt{a}\right)\left(\frac{b}{2\sqrt{a}}\right)x + \left(\frac{b}{2\sqrt{a}}\right)^2 =...
Can you write Python code for it?

# IMPORTANT, you should run second cell with import statements
# or current cell should contain
# import sympy
x, a, b, c = sympy.symbols('x a b c') # Define symbols for parameters
sympy.init_printing() # LaTeX-formatted result for printing
sympy.solve(a * x**2 + b * x + c, x) # solve parametric equation

import math
def solve_quadratic_equation(a, b, c):
    """
    Returns the real solutions of the quadratic equation ax^2 + bx + c = 0
    """
    # Check if we have linear equation, a = 0
    if a == 0: # if we do
        return [ -(c/b)] # return the single root
    # if not, we are continue with quadratic equation
    # determine the value of b**2 - 4ac
    d = float(b * b - 4.0 * a * c)
    if d < 0: # there is no roots
        return [] # return empty array
    else: #  we have some roots
        if d == 0: # only one root
            return [ (-b)/(2.0*a) ]
        else: # or two roots
            return [(-b - (math.sqrt(d)))/(2.0*a), (-b + (math.sqrt(d)))/(2.0*a)]

# Testing: Execute this cell. The outputs should match the expected outputs. Feel free to write more tests
print(solve_quadratic_equation(1, -2, -1.25)) # [-0.5, 2.5] <== parameters were changed to apply to the result in the comment
print(solve_quadratic_equation(1, -1, -2)) # original parameters
print(solve_quadratic_equation(1, -8, 16)) # [4.0]
print(solve_quadratic_equation(1, 1, 1)) # []
print(solve_quadratic_equation(0, 2, 5))


I am working on the file "MathConceptsForDevelopers-Sep17/04_Hight-SchoolMaths-E/High-School Maths Exercise/Solutions.ipynb".
The first task is:
Solution 5. Linearizing function
Try to linearize
$$ y = ae^{bx} $$
Can you write Python code for it?

from sympy import *
sympy.init_printing()
x, a, b = sympy.symbols('x, a, b', extended_real=True)
#y = sympy.Function('y')
y = sympy.symbols('y')
eqs = sympy.Eq(y, a * exp(b * x))
lne = sympy.Eq(log(eqs.lhs), expand_log(log(eqs.rhs), force=True))
lne


I am working on the file "MathConceptsForDevelopers-Sep17/04_Hight-SchoolMaths-E/High-School Maths Exercise/Solutions.ipynb".
The first task is:
Solution 7. Solving Equations Graphically
Plot multiple functions on one graph
Can you write Python code for it?

def plot_math_functions(functions, min_x, max_x, num_points):
    # Write your code here
    x = np.linspace(min_x, max_x, num_points)
    vectorized_fs = [np.vectorize(f) for f in functions]
    ys = [vectorized_f(x) for vectorized_f in vectorized_fs]
    fig, ax = plt.subplots() # set alias for pmatplotlib.pyplot.gca template for ploting
    # manipulate the plot object properties
    ax.spines["bottom"].set_position("zero") # insert the axis to zero
    ax.spines["left"].set_position("zero")
    ax.spines["top"].set_visible(False) # hide top frame
    ax.spines["right"].set_visible(False)
    for y in ys:
        plt.plot(x, y)
    ax.set_yticklabels(ax.get_yticks()) # extract ticks and set them to plt.subplots (ax)
    ax.set_xticklabels(ax.get_xticks())
    ylabels = ax.get_yticklabels() # create list of ticks labels
    xlabels = ax.get_xticklabels()
    # create variables for indexes from labels list where we are going to hide the labels
    xzero = 0
    yzero = 0
    # loop trough labels and extract index of text '0.0'
    for num, xlabel in enumerate(xlabels, start=0):
        if xlabel.get_text() == '0.0':
            xzero = num
    for num, ylabel in enumerate(ylabels, start=0):
        if ylabel.get_text() == '0.0':
            yzero = num
    yticks = ax.yaxis.get_major_ticks() # take all Y major ticks
    xticks = ax.xaxis.get_major_ticks()
    ax.set_xlabel('x') # we can have set_xlabel('name', fontsize = 12)
    ax.set_ylabel('y').set_rotation(0)
    # offset label for zero of X
    xticks[xzero].label1.set_horizontalalignment('right')
    # hide label1 on index with zero of Y
    yticks[yzero].label1.set_visible(False)
    ax.xaxis.set_label_coords(1.02, yzero / len(ylabels)) # fine positioning of x label
    ax.yaxis.set_label_coords((1 * abs(min_x))/((max_x) - (min_x)), 1.05) # possitioning of Y label
    plt.show()

plot_math_functions([lambda x: 2 * x + 3, lambda x: 0], -3, 5, 1000)
plot_math_functions([lambda x: 3 * x**2 - 2 * x + 5, lambda x: 3 * x + 7], -2, 3,...


I am working on the file "MathConceptsForDevelopers-Sep17/04_Hight-SchoolMaths-E/High-School Maths Exercise/Solutions.ipynb".
The first task is:
Solution 8. Trigonometric functions
Use the plotting function you wrote above to plot the inverse trigonometric functions
Can you write Python code for it?

plot_math_functions([lambda x: np.arcsin(x), lambda x: np.arccos(x), lambda x: np.arctan(x), lambda x: np.arctan(1/x)], -1.0, 1.0, 1000)


I am working on the file "MathNotes/LinearAlgebra.ipynb".
The first task is:
Column space

All linear combinations of columns

$ Ax = b $ is solvable if and only if b is in the column space of A

Nullspace

Solutions about Ax = 0
Solutions about Ax = b
Rank

It is the number of pivots

It can tell the number of independent columns

It is the dimension of column space, row space and nullspace

For example:
$ A = \begin{bmatrix}
       1 & 2 & 2 \
       2 & 4 & 1 \
       3 & 6 & 4 \
       \end{bmatrix}$
row2 - row1 * 2 : $\begin{bmatrix}
                    1 & 2 & 2 \
                    0 & 0 & -3 \
                    3 & 6 & 4 \
                    \end{bmatrix}$
row3 - row1 * 3 : $\begin{bmatrix}
                    1 & 2 & 2 \
                    0 & 0 & -3 \
                    0 & 0 & -2 \
                    \end{bmatrix}$
row3 - row2 * 3/2 : $ \begin{bmatrix}
                       1 & 2 & 2 \
                       0 & 0 & -3 \
                       0 & 0 & 0 \
                       \end{bmatrix} $
So, $ rank(A) = 2 $
Can you write Python code for it?


I am working on the file "MathNotes/LinearAlgebra.ipynb".
The first task is:
Gaussian Elimination

one way to solve $ Ax = b $

For example:
$ \begin{bmatrix}1 & 1 & 1 \ 3 & 1 & 2 \ 2 & 3 & 4 \ \end{bmatrix} \begin{bmatrix}x_1 \ x_2 \ x_3 \ \end{bmatrix} = \begin{bmatrix} 6 \ 11 \ 20 \ \end{bmatrix},\; A = \begin{bmatrix}1 & 1 & 1 \ 3 & 1 & 2 \ 2 & 3 & 4 \ \end{bmatrix},\; x = \begin{bmatrix}x_1 \ x_2 \ x_3 \ \end{bmatrix},\; b = \begin{bmatrix} 6 \ 11 \ 20 \ \end{bmatrix}$
$ [A, b] =  \begin{bmatrix}1 & 1 & 1 & 6\ 3 & 1 & 2 & 11\ 2 & 3 & 4 & 20\ \end{bmatrix}$
row2 - row1 * 3: $ \begin{bmatrix}1 & 1 & 1 & 6\ 0 & -2 & -1 & -7\ 2 & 3 & 4 & 20\ \end{bmatrix} $
row3 - row1 * 2: $ \begin{bmatrix}1 & 1 & 1 & 6\ 0 & -2 & -1 & -7\ 0 & 1 & 2 & 8\ \end{bmatrix} $
exchange row2 with row3: $ \begin{bmatrix}1 & 1 & 1 & 6\ 0 & 1 & 2 & 8\ 0 & -2 & -1 & -7\ \end{bmatrix} $
row1 - row2: $ \begin{bmatrix}1 & 0 & -1 & -2\ 0 & 1 & 2 & 8\ 0 & -2 & -1 & -7\ \end{bmatrix} $
row3 + row2 * 2: $ \begin{bmatrix}1 & 0 & -1 & -2\ 0 & 1 & 2 & 8\ 0 & 0 & 3 & 9\ \end{bmatrix} $
divide row3 by 3: $ \begin{bmatrix}1 & 0 & -1 & -2\ 0 & 1 & 2 & 8\ 0 & 0 & 1 & 3\ \end{bmatrix} $
row1 + row3: $ \begin{bmatrix}1 & 0 & 0 & 1\ 0 & 1 & 2 & 8\ 0 & 0 & 1 & 3\ \end{bmatrix} $
row2 - row3 * 2: $ \begin{bmatrix}1 & 0 & 0 & 1\ 0 & 1 & 0 & 2\ 0 & 0 & 1 & 3\ \end{bmatrix} $
$ x = \begin{bmatrix} x_1 \ x_2 \ x_3 \ \end{bmatrix} = \begin{bmatrix} 1 \ 2 \ 3 \ \end{bmatrix}$
Can you write Python code for it?


I am working on the file "MathNotes/LinearAlgebra.ipynb".
The first task is:
Projection Matrix
$ P = A(A^TA)^{-1}A^T$
Transpose

$A^T (i,j) = A(j,i)$

For example:
$ A = \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \ 7 & 8 & 9 \ \end{bmatrix} ,\; A^T = \begin{bmatrix}1 & 4 & 7 \ 2 & 5 & 8 \ 3 & 6 & 9 \ \end{bmatrix}$
Can you write Python code for it?


I am working on the file "Mathammer-Truncation.ipynb".
The first task is:
A completely different picture! The chance to kill outright is not that much higher, albeit it's now over 50%. The biggest difference is in the chance of making no progress at all, which went from ~15% to ~5%. What is driving this? It's the fact that the distribution of 12 shots with fixed damage is much 'narrower' than of 10 with d6 damage, there is less variance, which specifically means there is less of a tail to the left of 0, which in the Kataphron case is substantial.
Add in canticle for re-rolls of 1s:
Can you write Python code for it?

profiles[0]['p_hit'] = 1 / 2 + 1 / 6 * 1 / 2
wound_chart(profiles, targets)


I am working on the file "Mathammer-Truncation.ipynb".
The first task is:
That is certainly not a binomial distribution.
The plot reports an expecation, which is calculated from the actual distribution. Compare it with the naive calculation: 8 shots, 0.39 to hit, 0.67 to wound, 0.67 unsaved, d6 damage, take 3.5 expectation, giving:
Can you write Python code for it?


I am working on the file "Mathammer-Truncation.ipynb".
The first task is:
The expectation has grown by 0.5, but the chance of a kill increased by 10 percentage points, now it's a coin toss. Unfortunately, the Venom still has pretty good odds of escaping unscathed. Such is the power of Flickerfield and invulnerable save. 5 Kataphrons is 175 points, and clearly we are not getting efficiency here. Let's consider a unit that can actually hit, 3 Ironstriders (BS 3+) at 150 points with twin Autocannons (Heavy 4, S6, 2 damage), without canticle:
Can you write Python code for it?

profiles[0] = {'shots': 12, 'p_hit': 1 / 2, 'p_wound': 2 / 3, 'damage': '2'}
wound_chart(profiles, targets)


I am working on the file "Mathematical-Notation-Sums-and-Products.ipynb".
The first task is:
Instead of writing s = s + i we could have written s += i  (read this as "s is whatever it was before plus the value of i"). So we could rewrite that for loop as:
Can you write Python code for it?

x = [2,4,6,8,10]

s = 0
for i in x:
    s += i

print("The sum of x is:", s)


I am working on the file "Mathematical-Notation-Sums-and-Products.ipynb".
The first task is:
Now what if we wanted to calculate the sum of the reciprocals of each element of x?  A simple change to our code give us:
Can you write Python code for it?

x = [2,4,6,8,10]
s = 0
for i in x:
    s += (1/i)

print("The sum of the reciprocals of x is:", s)


I am working on the file "Mathhammer-Intro.ipynb".
The first task is:
A much stronger effect. You still cannot really rely on getting 3+ hits (~70% chance), but they will happen often enough and rarely would you get less than 2.
Now consider Veterans, who can shoot somewhat better and cost 6 points against 4 per a regular squaddie, what does that improvement in Ballistic Skill actually yield?
Can you write Python code for it?

profiles[0]['shots'] = 10
profiles[0]['p_hit'] = 2 / 3
wound_chart(damage_dealt(profiles[0]), profiles)


I am working on the file "Mathhammer-Intro.ipynb".
The first task is:
As you can see, the squad has about 47% chance of scoring 1 or 0 hits and around 53% chance of scoring 2 or more hits. The expectation is 1.7, which is misleading: the squad actually has better than even odds of getting 2+ hits. The biggest takeaway  is that you can only really count on getting 1 hit, and even then you will be frustrated every sixth roll.
Let us take a look at how orders affect things. First we put our squad under 'Take Aim!' increasing the to hit chance to 0.583:
Can you write Python code for it?

profiles[0]['p_hit'] = 0.583
wound_chart(damage_dealt(profiles[0]), profiles)


I am working on the file "Mathhammer-Intro.ipynb".
The first task is:
Maybe not as much as one would think, the chance to score 2+ has gone up to around 68%, compared to the troop squad's 53%. If you have 60 points to spend, you are probably better off with 15 regular shots:
Can you write Python code for it?

profiles[0]['shots'] = 15
profiles[0]['p_hit'] = 1 / 2
wound_chart(damage_dealt(profiles[0]), profiles)


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
Awww, nothing happened! This is because by default mpl will not show anything until told to do so, as we mentioned earlier in the "backend" discussion.
Instead, we'll need to call plt.show()
Can you write Python code for it?


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
Axes
All plotting is done with respect to an Axes. An Axes is made up of Axis objects and many other things. An Axes object must belong to a Figure (and only one Figure). Most commands you will ever issue will be with respect to this Axes object.
Typically, you'll set up a Figure, and then add an Axes to it.
You can use fig.add_axes, but in most cases, you'll find that adding a subplot will fit your needs perfectly. (Again a "subplot" is just an axes on a grid system.)
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111) # We'll explain the "111" later. Basically, 1 row and 1 column.
ax.set(xlim=[0.5, 4.5], ylim=[-2, 8], title='An Example Axes', ylabel='Y-Axis', xlabel='X-Axis')
plt.show()


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
Axes methods vs. pyplot
Interestingly, just about all methods of an Axes object exist as a function in the pyplot module (and vice-versa). For example, when calling plt.xlim(1, 10), pyplot calls ax.set_xlim(1, 10) on whichever Axes is "current". Here is an equivalent version of the above example using just pyplot.
Can you write Python code for it?


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
Clearly this can get repitive quickly.  Therefore, Matplotlib's set method can be very handy.  It takes each kwarg you pass it and tries to call the corresponding "setter".  For example, foo.set(bar='blah') would call foo.set_bar('blah').
Note that the set method doesn't just apply to Axes; it applies to more-or-less all matplotlib objects.
However, there are cases where you'll want to use things like ax.set_xlabel('Some Label', size=25) to control other options for a particular function.
Basic Plotting
Most plotting happens on an Axes.  Therefore, if you're plotting something on an axes, then you'll use one of its methods.
We'll talk about different plotting methods in more depth in the next section.  For now, let's focus on two methods: plot and scatter.
plot draws points with lines connecting them.  scatter draws unconnected points, optionally scaled or colored by additional variables.
As a basic example:
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot([1, 2, 3, 4], [10, 20, 25, 30], color='lightblue', linewidth=3)
ax.scatter([0.3, 3.8, 1.2, 2.5], [11, 25, 9, 26], color='darkgreen', marker='^')
ax.set_xlim(0.5, 4.5)
plt.show()


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
Figures
Now let's create a figure...
Can you write Python code for it?


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
Great, a blank figure! Not terribly useful yet.
However, while we're on the topic, you can control the size of the figure through the figsize argument, which expects a tuple of (width, height) in inches.
A really useful utility function is figaspect
Can you write Python code for it?

# Twice as tall as it is wide:
fig = plt.figure(figsize=plt.figaspect(2.0))
plt.show()


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
Matplotlib
Introduction
Matplotlib is a library for producing publication-quality figures. mpl (for short) was designed from the bottom-up to serve dual-purposes. First, to allow for interactive, cross-platform control of figures and plots, and second, to make it very easy to produce static raster or vector graphics files without the need for any GUIs. Furthermore, mpl -- much like Python itself -- gives the developer complete control over the appearance of their plots, while still being very usable through a powerful defaults system.
Online Documentation
The matplotlib.org project website is the primary online resource for the library's documentation. It contains examples, FAQs, API documentation, and, most importantly, the gallery.
Gallery
Many users of matplotlib are often faced with the question, "I want to make a plot that has X with Y in the same figure, but it needs to look like Z". Good luck getting an answer from a web search with that query. This is why the gallery is so useful, because it showcases the variety of ways one can make plots. Browse through the gallery, click on any figure that has pieces of what you want to see the code that generated it. Soon enough, you will be like a chef, mixing and matching components to produce your masterpiece!
As always, if you have a new and interesting plot that demonstrates a feature of matplotlib, feel free to submit a well-commented version of the example code for inclusion.
Mailing Lists and StackOverflow
When you are just simply stuck, and can not figure out how to get something to work, or just need some hints on how to get started, you will find much of the community at the matplotlib-users mailing list. This mailing list is an excellent resource of information with many friendly members who just love to help out newcomers. The number one rule to remember with this list is to be persistant. While many questions do get answered fairly quickly, some do fall through the cracks, or the one person who knows the answer isn't available. Therefore, try again...
Can you write Python code for it?

import matplotlib
print(matplotlib.__version__)
print(matplotlib.get_backend())


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
Much cleaner, and much clearer! So, why will most of my examples not follow the pyplot approach? Because PEP20 "The Zen of Python" says:
"Explicit is better than implicit"
While very simple plots, with short scripts would benefit from the conciseness of the pyplot implicit approach, when doing more complicated plots, or working within larger scripts, you will want to explicitly pass around the Axes and/or Figure object to operate upon.
The advantage of keeping which axes we're working with very clear in our code will become more obvious when we start to have multiple axes in one figure.
Multiple Axes
We've mentioned before that a figure can have more than one Axes on it.  If you want your axes to be on a regular grid system, then it's easiest to use plt.subplots(...) to create a figure and add the axes to it automatically.
For example:
Can you write Python code for it?

fig, axes = plt.subplots(nrows=2, ncols=2)
plt.show()


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
Notice the call to set. Matplotlib's objects typically have lots of "explicit setters" -- in other words, functions that start with set_<something> and control a particular option.
To demonstrate this (and as an example of IPython's tab-completion), try typing ax.set_ in a code cell, then hit the <Tab> key.  You'll see a long list of Axes methods that start with set.
For example, we could have written the third line above as:
Can you write Python code for it?

ax.set_xlim([0.5, 4.5])
ax.set_ylim([-2, 8])
ax.set_title('An Example Axes')
ax.set_ylabel('Y-Axis')
ax.set_xlabel('X-Axis')


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
On with the show!
Matplotlib is a large project and can seem daunting at first. However, by learning the components, it should begin to feel much smaller and more approachable.
Anatomy of a "Plot"
People use "plot" to mean many different things.  Here, we'll be using a consistent terminology (mirrored by the names of the underlying classes, etc):
<img src="images/figure_axes_axis_labeled.png">
The Figure is the top-level container in this hierarchy.  It is the overall window/page that everything is drawn on.  You can have multiple independent figures and Figures can contain multiple Axes.
Most plotting ocurs on an Axes.  The axes is effectively the area that we plot data on and any ticks/labels/etc associated with it.  Usually we'll set up an Axes with a call to subplot (which places Axes on a regular grid), so in most cases, Axes and Subplot are synonymous.
Each Axes has an XAxis and a YAxis.  These contain the ticks, tick locations, labels, etc.  In this tutorial, we'll mostly control ticks, tick labels, and data limits through other mechanisms, so we won't touch the individual Axis part of things all that much.  However, it's worth mentioning here to explain where the term Axes comes from.
Getting Started
In this tutorial, we'll use the following import statements. These abbreviations are semi-standardized, and most tutorials, other scientific python code, etc that you'll find elsewhere will use them as well.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt


I am working on the file "MatplotLib Pynotebooks/AnatomyOfMatplotlib-Part1-Figures_Subplots_and_layouts.ipynb".
The first task is:
plt.subplots(...) created a new figure and added 4 subplots to it.  The axes object that was returned is a 2D numpy object array.  Each item in the array is one of the subplots.  They're laid out as you see them on the figure.
Therefore, when we want to work with one of these axes, we can index the axes array and use that item's methods.
For example:
Can you write Python code for it?

fig, axes = plt.subplots(nrows=2, ncols=2)
axes[0,0].set(title='Upper Left')
axes[0,1].set(title='Upper Right')
axes[1,0].set(title='Lower Left')
axes[1,1].set(title='Lower Right')

# To iterate over all items in a multidimensional numpy array, use the `flat` attribute
for ax in axes.flat:
    # Remove all xticks and yticks...
    ax.set(xticks=[], yticks=[])

plt.show()