I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The feature
The feature is based on a simple zero crossing rate (zcr). We choose to only keep the increasing crossing. I don't think it is making a lot of differences for the final performance of the keyword recognition.
The zcr function is computing the zcr for a window of samples.
Can you write Python code for it?

def zcr(w):
    w = w-np.mean(w)
    f=w[:-1]
    g=w[1:]
    k=np.count_nonzero(np.logical_and(f*g<0, g>f))
    return(1.0*k/len(f))


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The feature code is working, so now we can implement the predict:
Can you write Python code for it?

def dsp_predict(feat):

    res=dsp.arm_dot_prod_f32(coef_f32,feat)
    res = res + intercept_f32

    if res[0]<0:
        return(-1)
    else:
        return(0)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The final feature is the zcr computed on a segment of 1 second and filtered. We are using a sliding window and using a Hann window.
Can you write Python code for it?

def feature(data):
    samplerate=16000
    input_len = 16000

    # The speech pattern is padded to ensure it has a duration of 1 second

    waveform = data[:input_len]

    zero_padding = np.zeros(
      16000 - waveform.shape[0],
      dtype=np.float32)

    signal = np.hstack([waveform, zero_padding])

    # We decompose the intput signal into overlapping window. And the signal in each window
    # is premultiplied by a Hann window of the right size.
    # Warning : if you change the window duration and audio offset, you'll need to change the value
    # in the scripts used for the scheduling of the compute graph later.
    winDuration=25e-3
    audioOffsetDuration=10e-3
    winLength=int(np.floor(samplerate*winDuration))
    audioOffset=int(np.floor(samplerate*audioOffsetDuration))
    overlap=winLength-audioOffset
    window=hann(winLength,sym=False)
    reta=[zcr(x*window) for x in sliding_window_view(signal,winLength)[::audioOffset,:]]

    # The final signal is filtered. We have tested several variations on the feature. This filtering is
    # improving the recognition
    reta=scipy.signal.lfilter(np.ones(10)/10.0,[1],reta)
    return(np.array(reta))


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The implementation of the nodes is in kws/AppNodes.h. It is very similar to the appnodes.py but using the CMSIS-DSP C API.
The C++ template are used only to minimize the overhead at runtime.
Arduino
You need to have the arduino command line tools installed. And they need to be in your PATH so that the notebook can find the tool.
We are using the Arduino Nano 33 BLE
Building and upload
Can you write Python code for it?

!arduino-cli board list

!arduino-cli config init

!arduino-cli lib install Arduino_CMSIS-DSP


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The patterns
The below class is representing a Pattern. A pattern can either be a sound file from the TensorFlow Lite examples and in this case we compute the feature and the label id.
The pattern can also be a random white noise. In that case, we also compute the feature and the class id is -1.
Note that when you use the signal property, the speech patterns will return the content of the file but noise patterns will generate a random noise which will thus be different each time.
Can you write Python code for it?

class Pattern:
    def __init__(self,p):
        global UNKNOWN_CLASS
        if isinstance(p, str):
            self._isFile=True
            self._filename=p
            self._label=get_label_id(p)

            data, samplerate = sf.read(self._filename)
            self._feature = feature(data)
        else:
            self._isFile=False
            self._noiseLevel=p
            self._label=UNKNOWN_CLASS

            noise=np.random.randn(16000)*p
            self._feature=feature(noise)

    @property
    def label(self):
        return(self._label)

    @property
    def feature(self):
        return(self._feature)

    # Only useful for plotting
    # The random pattern will be different each time
    @property
    def signal(self):
        if not self._isFile:
            return(np.random.randn(16000)*self._noiseLevel)
        else:
            data, samplerate = sf.read(self._filename)
            return(data)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The score is as good as the F32 implementation.
Q15 Implementation
It is the same as Q31 but using Q15 functions.
Can you write Python code for it?

scaled_coef=clfb.best_estimator_.coef_
coef_shift=0
while np.max(np.abs(scaled_coef)) > 1:
    scaled_coef = scaled_coef / 2.0
    coef_shift = coef_shift + 1

coef_q15=fix.toQ15(scaled_coef)

scaled_intercept = clfb.best_estimator_.intercept_
intercept_shift = 0
while np.abs(scaled_intercept) > 1:
    scaled_intercept = scaled_intercept / 2.0
    intercept_shift = intercept_shift + 1

intercept_q15=fix.toQ15(scaled_intercept)

def dsp_zcr_q15(w):
    m = dsp.arm_mean_q15(w)
    # Negate can saturate so we use CMSIS-DSP function which is working on array (and we have a scalar)
    m = dsp.arm_negate_q15(np.array([m]))[0]
    w = dsp.arm_offset_q15(w,m)

    f=w[:-1]
    g=w[1:]
    k=np.count_nonzero(np.logical_and(np.logical_or(np.logical_and(f>0,g<0), np.logical_and(f<0,g>0)),g>f))

    # k < len(f) so shift should be 0 except when k == len(f)
    # When k==len(f) normally quotient is 0x4000 and shift 1 and we convert
    # this to 0x7FFF and shift 0
    status,quotient,shift_val=dsp.arm_divide_q15(k,len(f))
    if shift_val==1:
        return(dsp.arm_shift_q15(np.array([quotient]),shift)[0])
    else:
        return(quotient)

firq15 = dsp.arm_fir_instance_q15()

def dsp_feature_q15(data):
    samplerate=16000
    input_len = 16000

    waveform = data[:input_len]

    zero_padding = np.zeros(
      16000 - waveform.shape[0],
      dtype=np.int16)

    signal = np.hstack([waveform, zero_padding])

    winDuration=25e-3
    audioOffsetDuration=10e-3
    winLength=int(np.floor(samplerate*winDuration))
    audioOffset=int(np.floor(samplerate*audioOffsetDuration))
    overlap=winLength - audioOffset

    window=fix.toQ15(hann(winLength,sym=False))
    reta=[dsp_zcr_q15(dsp.arm_mult_q15(x,window)) for x in sliding_window_view(signal,winLength)[::audioOffset,:]]

    # Reset state and filter
    blockSize=98
    numTaps=10
    stateLength = numTaps + blockSize - 1
    dsp.arm_fir_init_q15(firq15,10,fix.toQ15(np.ones(10)/10.0),np.zeros(stateLength,dtype=np.int16))
 ...


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The speech commands
We are using the simplified speech commands from the TensorFlow Lite tutorial.
Those commands can be downloaded from this link: "http://storage.googleapis.com/download.tensorflow.org/data/mini_speech_commands.zip"
Once the zip has been uncompressed, you'll need to change the path of the folder below.
The below code is loading the list of commands available in the mini_speech_commands folder and it is describing the words we want to detect. Here we only want to detect the Yes keyword.
You can add other keywords but the CMSIS-DSP implementation in this notebook is only supporting one keyword.
Nevertheless, if you'd like to experiment with the training of the ML model and different features, then you can work with more commands.
Can you write Python code for it?

MINISPEECH="mini_speech_commands"
commands=np.array([os.path.basename(f) for f in glob.glob(os.path.join(MINISPEECH,"mini_speech_commands","*"))])
commands=commands[commands != "README.md"]
# Any other word will be recognized as unknown
to_keep=['yes']


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
To describe our compute graph, we need to describe the nodes which are used in this graph.
Each node is described by its inputs and outputs. For each IO, we define the data type and the number of samples read or written on the IO.
We need the following nodes in our system:
* A source to get audio samples from the Arduino PDM driver
* A Sink to generate the messages on the Arduino serial port
* A feature node to compute the feature for a given window (and pre-multipliying with the Hann window)
* A FIR node which is filtering all the features for one second of signal
* A KWS node which is doing the logistic regression
In addition to that we need sliding windows:
* Like in the Python code above we need a sliding window for audio
* After each recognition attempt on a segment of 1 second, we want to slide the recognition window by 0.5 seconds. It can be implemented with a sliding window on the features before the FIR
Can you write Python code for it?

class Source(GenericSource):
    def __init__(self,name,inLength):
        GenericSource.__init__(self,name)
        q15Type=CType(Q15)
        self.addOutput("o",q15Type,inLength)

    @property
    def typeName(self):
        return "Source"

class Sink(GenericSink):
    def __init__(self,name,outLength):
        GenericSink.__init__(self,name)
        q15Type=CType(Q15)
        self.addInput("i",q15Type,outLength)

    @property
    def typeName(self):
        return "Sink"

class Feature(GenericNode):
    def __init__(self,name,inLength):
        GenericNode.__init__(self,name)

        q15Type=CType(Q15)
        self.addInput("i",q15Type,inLength)
        self.addOutput("o",q15Type,1)

    @property
    def typeName(self):
        return "Feature"

class FIR(GenericNode):
    def __init__(self,name,inLength,outLength):
        GenericNode.__init__(self,name)

        q15Type=CType(Q15)
        self.addInput("i",q15Type,inLength)
        self.addOutput("o",q15Type,outLength)

    @property
    def typeName(self):
        return "FIR"

class KWS(GenericNode):
    def __init__(self,name,inLength):
        GenericNode.__init__(self,name)

        q15Type=CType(Q15)
        self.addInput("i",q15Type,inLength)
        self.addOutput("o",q15Type,1)

    @property
    def typeName(self):
        return "KWS"


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
We are getting very similar results to the reference implementation. Now let's explore fixed point.
Q31 implementation
First thing to do is to convert the F32 values of the ML mode into Q31.
But we need values in [-1,1]. So we rescale those values and keep track of the shift required to restore the original value.
Then, we convert those rescaled values to Q31.
Can you write Python code for it?

scaled_coef=clfb.best_estimator_.coef_
coef_shift=0
while np.max(np.abs(scaled_coef)) > 1:
    scaled_coef = scaled_coef / 2.0
    coef_shift = coef_shift + 1

coef_q31=fix.toQ31(scaled_coef)

scaled_intercept = clfb.best_estimator_.intercept_
intercept_shift = 0
while np.abs(scaled_intercept) > 1:
    scaled_intercept = scaled_intercept / 2.0
    intercept_shift = intercept_shift + 1

intercept_q31=fix.toQ31(scaled_intercept)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
We are using the best estimator found during the randomized search:
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
We can now save the model so that next time we want to play with the notebook and test the CMSIS-DSP implementation we do not have to retrain the model:
Can you write Python code for it?

with open("logistic.pickle","wb") as f:
    s = pickle.dump(clfb,f)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
We compute the final score. 0.8 is really the minimum acceptable value for this kind of demo.
With the zcr feature, if you try to detect Yes, No, Unknown, you'll get a score of around 0.6 which is very bad.
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
We need some parameters. Those parameters need to be coherent with the values defined in the features in the above code.
AUDIO_INTERRUPT_LENGTH is the audio length generated by the source. But it is not the audio length generated by th PDM driver on Arduino. The Arduino implementation of the Source is doing the adaptation as we will see below.
Can you write Python code for it?

q15Type=CType(Q15)
FS=16000
winDuration=25e-3
audioOffsetDuration=10e-3
winLength=int(np.floor(FS*winDuration))
audio_input_length=int(np.floor(FS*audioOffsetDuration))
AUDIO_INTERRUPT_LENGTH = audio_input_length


I am working on the file "CNN Lab.ipynb".
The first task is:
Evaluation of the model
Once you are completely satisfied with your model, evaluate the performance of the model on the test set.
Can you write Python code for it?

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))

    test_accuracy = evaluate(X_test, y_test)
    print("Test Accuracy = {:.3f}".format(test_accuracy))


I am working on the file "CNN Lab.ipynb".
The first task is:
Implementing LeNet-5
Implement the LeNet-5 neural network architecture.
Input
The LeNet architecture accepts a 32x32xC image as input, where C is the number of color channels. Since MNIST images are grayscale, C is 1 in this case.
Architecture
Layer 1: Convolutional. The output shape should be 28x28x6.
Activation. Your choice of activation function.
Pooling. The output shape should be 14x14x6.
Layer 2: Convolutional. The output shape should be 10x10x16.
Activation. Your choice of activation function.
Pooling. The output shape should be 5x5x16.
Flatten. Flatten the output shape of the final pooling layer such that it's 1D instead of 3D. The easiest way to do is by using tf.contrib.layers.flatten, which is already imported for you.
Layer 3: Fully Connected. This should have 120 outputs.
Activation. Your choice of activation function.
Layer 4: Fully Connected. This should have 84 outputs.
Activation. Your choice of activation function.
Layer 5: Fully Connected (Logits). This should have 10 outputs.
Output
Return the result of the 2nd fully connected layer.
Can you write Python code for it?

def LeNet(x):
    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer
    mu = 0
    sigma = 0.1

    # TODO: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.

    # TODO: Activation.

    # TODO: Pooling. Input = 28x28x6. Output = 14x14x6.

    # TODO: Layer 2: Convolutional. Output = 10x10x16.

    # TODO: Activation.

    # TODO: Pooling. Input = 10x10x16. Output = 5x5x16.

    # TODO: Flatten. Input = 5x5x16. Output = 400.

    # TODO: Layer 3: Fully Connected. Input = 400. Output = 120.

    # TODO: Activation.

    # TODO: Layer 4: Fully Connected. Input = 120. Output = 84.

    # TODO: Activation.

    # TODO: Layer 5: Fully Connected. Input = 84. Output = 10.

    return logits


I am working on the file "CNN Lab.ipynb".
The first task is:
Training pipeline
Can you write Python code for it?

x = tf.placeholder(tf.float32, (None, 32, 32, 1))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 10)

learning_rate = 0.001
epochs = 10
batch_size = 128

# the model you defined
logits = LeNet(x)

# define the evaluation functions
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)

loss_operation = tf.reduce_mean(cross_entropy)

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)

training_operation = optimizer.minimize(loss_operation)

correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))

accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

saver = tf.train.Saver()

def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, batch_size):
        batch_x, batch_y = X_data[offset:offset + batch_size], y_data[offset:offset + batch_size]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples


I am working on the file "CNN training & evaluation - intestinal parasites.ipynb".
The first task is:
CNN training
Can you write Python code for it?

def CNN(n_epochs):
    net1 = NeuralNet(
        layers=[
        ('input', layers.InputLayer),
        ('conv1', layers.Conv2DLayer),      #Convolutional layer.  Params defined below
        ('pool1', layers.MaxPool2DLayer),   # Like downsampling, for execution speed
        ('conv2', layers.Conv2DLayer),
        ('hidden3', layers.DenseLayer),
        ('output', layers.DenseLayer),
        ],

    input_shape=(None, 3, opts['patch_size'][0]/opts['image_downsample'],
                 opts['patch_size'][0]/opts['image_downsample']),
    conv1_num_filters=7,
    conv1_filter_size=(5, 5),
    conv1_nonlinearity=lasagne.nonlinearities.rectify,

    pool1_pool_size=(2, 2),

    conv2_num_filters=12,
    conv2_filter_size=(2, 2),
    conv2_nonlinearity=lasagne.nonlinearities.rectify,

    hidden3_num_units=500,
    output_num_units=2,
    output_nonlinearity=lasagne.nonlinearities.softmax,

    update_learning_rate=0.0001,
    update_momentum=0.9,

    max_epochs=n_epochs,
    verbose=1,
    )
    return net1

cnn = CNN(50).fit(train_X, train_y)


I am working on the file "CNN training & evaluation - intestinal parasites.ipynb".
The first task is:
Evaluation: compare with classification based on morphological feature extraction
Can you write Python code for it?

featureset = [3,7,11,12,15,17]
centiles = [0,25,50,75,100]

pb = ProgressBar(train_X.shape[0])
train_X_f = []
for i in range(train_X.shape[0]):
    if i % 100 == 0:
        pb.step(i)
    graypatch = cv2.cvtColor(np.swapaxes(train_X[i,:,:,:],0,2).astype('uint8'), cv2.COLOR_BGR2GRAY)
    train_X_f.append(shapefeatures.extract(graypatch,attributes=featureset,centiles=centiles, momentfeatures=True))
train_X_f = np.vstack(train_X_f)

test_X_f = []
for i in range(test_X.shape[0]):
    if i % 100 == 0:
        pb.step(i)
    graypatch = cv2.cvtColor(np.swapaxes(test_X[i,:,:,:],0,2).astype('uint8'), cv2.COLOR_BGR2GRAY)
    test_X_f.append(shapefeatures.extract(graypatch,attributes=featureset,centiles=centiles, momentfeatures=True))
test_X_f = np.vstack(test_X_f)

clf = ensemble.ExtraTreesClassifier(n_estimators=100, max_depth=5, n_jobs=-1)
clf.fit(train_X_f, train_y)
y_pred_CLF = clf.predict_proba(test_X_f)

false_positive_rate_CNN, true_positive_rate_CNN, thresholds_CNN = metrics.roc_curve(test_y, y_pred[:,1])
roc_auc_CNN = metrics.auc(false_positive_rate_CNN, true_positive_rate_CNN)

precision_CNN, recall_CNN, thresholds_CNN = metrics.precision_recall_curve(test_y, y_pred[:,1])
average_precision_CNN = metrics.average_precision_score(test_y, y_pred[:, 1])

false_positive_rate_CLF, true_positive_rate_CLF, thresholds_CLF = metrics.roc_curve(test_y, y_pred_CLF[:,1])
roc_auc_CLF = metrics.auc(false_positive_rate_CLF, true_positive_rate_CLF)

precision_CLF, recall_CLF, thresholds_CLF = metrics.precision_recall_curve(test_y, y_pred_CLF[:,1])
average_precision_CLF = metrics.average_precision_score(test_y, y_pred_CLF[:, 1])

subplot(211)
plt.title('ROC' )
plt.plot(false_positive_rate_CNN, true_positive_rate_CNN, 'b', label='CNN: AUC=%.2f' % (roc_auc_CNN))
plt.plot(false_positive_rate_CLF, true_positive_rate_CLF, 'k--', label='ERT: AUC=%.2f' % (roc_auc_CLF))
plt.legend(loc='lower right')
plt.ylim([-.05, 1.05])
plt.xlim([-.05, 1.0])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive...


I am working on the file "CNN training & evaluation - intestinal parasites.ipynb".
The first task is:
Examine mistakes to understand network performance: false positives.
Find the negative-labelled patches with highest prediction score
Can you write Python code for it?

neg_indices = np.where(test_y==0)[0]
neg_scores = y_pred[neg_indices,1]
neg_indices = neg_indices[neg_scores.argsort()]
neg_indices = neg_indices[::-1]

neg_scores = y_pred[neg_indices,1]

N_samples_to_display = 12
offset = 55
for i in range(N_samples_to_display,2*N_samples_to_display):
    plt.subplot(2,N_samples_to_display,i+1)
    example_neg = test_X[neg_indices[i+offset],:,:,:]
    example_neg = np.swapaxes(example_neg,0,2)
    plt.imshow(example_neg)
    plt.title('%.3f' % neg_scores[i+offset])
    plt.tick_params(axis='both', which='both', bottom='off', top='off', labelbottom='off', right='off', left='off', labelleft='off')

plt.gcf().set_size_inches(1.5*N_samples_to_display,3)

plt.savefig('figs/plasmodium-falsedetections.png', bbox_inches='tight')


I am working on the file "CNN training & evaluation - intestinal parasites.ipynb".
The first task is:
Example of objects detected in an entire image
The white boxes represent annotations in the training data. Red boxes are detections by the convnet.
Can you write Python code for it?

reload(det)

fname = testfiles[12]
imfile = opts['img_dir'] + fname
opts['detection_threshold'] = 0.5
found = det.detect(imfile, cnn, opts)

im = misc.imread(imfile)

plt.box(False)
plt.xticks([])
plt.yticks([])

annofile = opts['annotation_dir'] + fname[:-3] + 'xml'
bboxes = readdata.get_bounding_boxes_for_single_image(annofile)
for bb in bboxes:
    bb = bb.astype(int)
    cv2.rectangle(im, (bb[0],bb[2]), (bb[1],bb[3]), (255,255,255), 2)

for f in found:
    f = f.astype(int)
    cv2.rectangle(im, (f[0],f[1]), (f[2],f[3]), (255,0,0), 2)

plt.gcf().set_size_inches(10,10)
plt.title('Detected objects in %s' % (imfile))
plt.imshow(im)

#cv2.imwrite('detectionimages/detected-' + os.path.basename(imfile),im)


I am working on the file "CNN training & evaluation - intestinal parasites.ipynb".
The first task is:
See highest-scored test patches
Can you write Python code for it?

prob_range = [.95,1.]

tmp_scores = y_pred.copy()[:,1]
tmp_scores[tmp_scores<prob_range[0]] = -1
tmp_scores[tmp_scores>prob_range[1]] = -1

pos_indices = tmp_scores.argsort()
pos_indices = pos_indices[::-1]

N_samples_to_display = 12
offset = 0
for i in range(N_samples_to_display,2*N_samples_to_display):
    plt.subplot(2,N_samples_to_display,i+1)
    example_neg = test_X[pos_indices[i+offset],:,:,:]
    example_neg = np.swapaxes(example_neg,0,2)
    plt.imshow(example_neg)
    plt.title('%.3f' % (tmp_scores[pos_indices[i+offset]]))
    plt.tick_params(axis='both', which='both', bottom='off', top='off', labelbottom='off', right='off', left='off', labelleft='off')

plt.gcf().set_size_inches(1.5*N_samples_to_display,3)

plt.savefig('figs/plasmodium-detectedpatches.png', bbox_inches='tight')


I am working on the file "CNN training & evaluation - intestinal parasites.ipynb".
The first task is:
See lowest scored test patches
Can you write Python code for it?

pos_indices = y_pred[:,1].argsort()

N_samples_to_display = 12

for i in range(N_samples_to_display,2*N_samples_to_display):
    plt.subplot(2,N_samples_to_display,i+1)
    example_neg = test_X[pos_indices[i],:,:,:]
    example_neg = np.swapaxes(example_neg,0,2)
    plt.imshow(example_neg)
    plt.title('%.3f' % (y_pred[pos_indices[i],1]))
    plt.tick_params(axis='both', which='both', bottom='off', top='off', labelbottom='off', right='off', left='off', labelleft='off')

plt.gcf().set_size_inches(1.5*N_samples_to_display,3)

plt.savefig('figs/plasmodium-testpatches-lowprob.png', bbox_inches='tight')


I am working on the file "CNN training & evaluation - intestinal parasites.ipynb".
The first task is:
View a random selection of positive and negative patches to see if they look right
Can you write Python code for it?

N_samples_to_display = 10
pos_indices = np.where(train_y)[0]
pos_indices = pos_indices[np.random.permutation(len(pos_indices))]
for i in range(N_samples_to_display):
    plt.subplot(2,N_samples_to_display,i+1)
    example_pos = train_X[pos_indices[i],:,:,:]
    example_pos = np.swapaxes(example_pos,0,2)
    plt.imshow(example_pos)
    plt.tick_params(axis='both', which='both', bottom='off', top='off', labelbottom='off', right='off', left='off', labelleft='off')

neg_indices = np.where(train_y==0)[0]
neg_indices = neg_indices[np.random.permutation(len(neg_indices))]
for i in range(N_samples_to_display,2*N_samples_to_display):
    plt.subplot(2,N_samples_to_display,i+1)
    example_neg = train_X[neg_indices[i],:,:,:]
    example_neg = np.swapaxes(example_neg,0,2)
    plt.imshow(example_neg)
    plt.tick_params(axis='both', which='both', bottom='off', top='off', labelbottom='off', right='off', left='off', labelleft='off')
plt.gcf().set_size_inches(1.5*N_samples_to_display,3)


I am working on the file "CNN training & evaluation - plasmodium (motic).ipynb".
The first task is:
Evaluation: compare with classification based on morphological feature extraction
Can you write Python code for it?


I am working on the file "CNN training & evaluation - plasmodium (motic).ipynb".
The first task is:
Examine mistakes to understand network performance: false positives.
Find the negative-labelled patches with highest prediction score
Can you write Python code for it?


I am working on the file "CNN training & evaluation - plasmodium (motic).ipynb".
The first task is:
See lowest scored test patches
Can you write Python code for it?


I am working on the file "CNN training & evaluation - plasmodium (phone).ipynb".
The first task is:
Evaluation: compare with classification based on morphological feature extraction
Can you write Python code for it?


I am working on the file "CNN training & evaluation - plasmodium (phone).ipynb".
The first task is:
Example of objects detected in an entire image
The white boxes represent annotations in the training data. Red boxes are detections by the convnet.
Can you write Python code for it?

reload(det)

fname = testfiles[1]
imfile = opts['img_dir'] + fname
opts['detection_probability_threshold'] = 0.95

found = det.detect(imfile, cnn, opts)

im = misc.imread(imfile)

plt.box(False)
plt.xticks([])
plt.yticks([])

annofile = opts['annotation_dir'] + fname[:-3] + 'xml'
bboxes = readdata.get_bounding_boxes_for_single_image(annofile)
for bb in bboxes:
    bb = bb.astype(int)
    cv2.rectangle(im, (bb[0],bb[2]), (bb[1],bb[3]), (255,255,255), 2)

for f in found:
    f = f.astype(int)
    cv2.rectangle(im, (f[0],f[1]), (f[2],f[3]), (255,0,0), 2)

plt.gcf().set_size_inches(10,10)
plt.title('Detected objects in %s' % (imfile))
plt.imshow(im)

#cv2.imwrite('detectionimages/detected-' + os.path.basename(imfile),im)


I am working on the file "CNN training & evaluation - plasmodium (phone).ipynb".
The first task is:
Make predictions and evaluate on test data
Can you write Python code for it?

y_pred = cnn.predict_proba(test_X)

false_positive_rate, true_positive_rate, thresholds = metrics.roc_curve(test_y, y_pred[:,1])
roc_auc = metrics.auc(false_positive_rate, true_positive_rate)

precision, recall, thresholds = metrics.precision_recall_curve(test_y, y_pred[:,1])
average_precision = metrics.average_precision_score(test_y, y_pred[:, 1])

subplot(121)
plt.title('ROC: AUC = %0.2f'% roc_auc)
plt.plot(false_positive_rate, true_positive_rate, 'b')
plt.legend(loc='lower right')
plt.plot([0,1],[0,1],'r--')
plt.ylim([-.05, 1.05])
plt.xlim([-.05, 1.0])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')

subplot(122)
plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('Precision-Recall: AP={0:0.2f}'.format(average_precision))
plt.legend(loc="lower left")

plt.gcf().set_size_inches(10,4)

#plt.savefig('figs/plasmodium-patchevaluation.png', bbox_inches='tight')

false_positive_rate, true_positive_rate, thresholds = metrics.roc_curve(test_y, y_pred[:,1])
true_positive_rate.shape, thresholds.shape
plt.plot(true_positive_rate, thresholds,label='True positive rate')
plt.plot(false_positive_rate, thresholds, label='False positive rate')
plt.xlabel('Threshold')
plt.legend(loc='upper left')


I am working on the file "CNN training & evaluation - plasmodium (phone).ipynb".
The first task is:
View a random selection of positive and negative patches to see if they look right
Can you write Python code for it?

29416./261345

N_samples_to_display = 10
pos_indices = np.where(train_y)[0]
pos_indices = pos_indices[np.random.permutation(len(pos_indices))]
for i in range(N_samples_to_display):
    plt.subplot(2,N_samples_to_display,i+1)
    example_pos = train_X[pos_indices[i],:,:,:]
    example_pos = np.swapaxes(example_pos,0,2)
    plt.imshow(example_pos)
    plt.tick_params(axis='both', which='both', bottom='off', top='off', labelbottom='off', right='off', left='off', labelleft='off')

neg_indices = np.where(train_y==0)[0]
neg_indices = neg_indices[np.random.permutation(len(neg_indices))]
for i in range(N_samples_to_display,2*N_samples_to_display):
    plt.subplot(2,N_samples_to_display,i+1)
    example_neg = train_X[neg_indices[i],:,:,:]
    example_neg = np.swapaxes(example_neg,0,2)
    plt.imshow(example_neg)
    plt.tick_params(axis='both', which='both', bottom='off', top='off', labelbottom='off', right='off', left='off', labelleft='off')
plt.gcf().set_size_inches(1.5*N_samples_to_display,3)


I am working on the file "CNN training & evaluation - tuberculosis.ipynb".
The first task is:
CNN training
Can you write Python code for it?

def CNN(n_epochs):
    net1 = NeuralNet(
        layers=[
        ('input', layers.InputLayer),
        ('conv1', layers.Conv2DLayer),      #Convolutional layer.  Params defined below
        ('pool1', layers.MaxPool2DLayer),   # Like downsampling, for execution speed
        ('conv2', layers.Conv2DLayer),
        ('hidden3', layers.DenseLayer),
        ('output', layers.DenseLayer),
        ],

    input_shape=(None, 3, opts['patch_size'][0]/opts['image_downsample'],
                 opts['patch_size'][0]/opts['image_downsample']),
    conv1_num_filters=7,
    conv1_filter_size=(3, 3),
    conv1_nonlinearity=lasagne.nonlinearities.rectify,

    pool1_pool_size=(2, 2),

    conv2_num_filters=12,
    conv2_filter_size=(2, 2),
    conv2_nonlinearity=lasagne.nonlinearities.rectify,

    hidden3_num_units=100,
    output_num_units=2,
    output_nonlinearity=lasagne.nonlinearities.softmax,

    update_learning_rate=0.0001,
    update_momentum=0.9,

    max_epochs=n_epochs,
    verbose=1,
    )
    return net1

cnn = CNN(500).fit(train_X, train_y)


I am working on the file "CNN training & evaluation - tuberculosis.ipynb".
The first task is:
Evaluation: compare with classification based on morphological feature extraction
Can you write Python code for it?


I am working on the file "CNN training & evaluation - tuberculosis.ipynb".
The first task is:
Examine mistakes to understand network performance: false positives.
Find the negative-labelled patches with highest prediction score
Can you write Python code for it?

neg_indices = np.where(test_y==0)[0]
neg_scores = y_pred[neg_indices,1]
neg_indices = neg_indices[neg_scores.argsort()]
neg_indices = neg_indices[::-1]

neg_scores = y_pred[neg_indices,1]

N_samples_to_display = 12
offset = 55
for i in range(N_samples_to_display,2*N_samples_to_display):
    plt.subplot(2,N_samples_to_display,i+1)
    example_neg = test_X[neg_indices[i+offset],:,:,:]
    example_neg = np.swapaxes(example_neg,0,2)
    plt.imshow(example_neg)
    plt.title('%.3f' % neg_scores[i+offset])
    plt.tick_params(axis='both', which='both', bottom='off', top='off', labelbottom='off', right='off', left='off', labelleft='off')

plt.gcf().set_size_inches(1.5*N_samples_to_display,3)

plt.savefig('figs/tuberculosis-falsedetections.png', bbox_inches='tight')


I am working on the file "CNN training & evaluation - tuberculosis.ipynb".
The first task is:
Example of objects detected in an entire image
The white boxes represent annotations in the training data. Red boxes are detections by the convnet.
Can you write Python code for it?

reload(det)

fname = testfiles[0]
imfile = opts['img_dir'] + fname
opts['detection_probability_threshold'] = 0.95
found = det.detect(imfile, cnn, opts)

im = misc.imread(imfile)

plt.box(False)
plt.xticks([])
plt.yticks([])

annofile = opts['annotation_dir'] + fname[:-3] + 'xml'
bboxes = readdata.get_bounding_boxes_for_single_image(annofile)
for bb in bboxes:
    bb = bb.astype(int)
    cv2.rectangle(im, (bb[0],bb[2]), (bb[1],bb[3]), (255,255,255), 8)

for f in found:
    f = f.astype(int)
    cv2.rectangle(im, (f[0],f[1]), (f[2],f[3]), (255,0,0), 8)

plt.gcf().set_size_inches(10,10)
plt.title('Detected objects in %s' % (imfile))
plt.imshow(im)

#cv2.imwrite('detectionimages/detected-' + os.path.basename(imfile),im)


I am working on the file "CNN training & evaluation - tuberculosis.ipynb".
The first task is:
Make predictions and evaluate on test data
Can you write Python code for it?

y_pred = cnn.predict_proba(test_X)

false_positive_rate, true_positive_rate, thresholds = metrics.roc_curve(test_y, y_pred[:,1])
roc_auc = metrics.auc(false_positive_rate, true_positive_rate)

precision, recall, thresholds = metrics.precision_recall_curve(test_y, y_pred[:,1])
average_precision = metrics.average_precision_score(test_y, y_pred[:, 1])

subplot(121)
plt.title('ROC: AUC = %0.2f'% roc_auc)
plt.plot(false_positive_rate, true_positive_rate, 'b')
plt.legend(loc='lower right')
plt.plot([0,1],[0,1],'r--')
plt.ylim([-.05, 1.05])
plt.xlim([-.05, 1.0])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')

subplot(122)
plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('Precision-Recall: AP={0:0.2f}'.format(average_precision))
plt.legend(loc="lower left")

plt.gcf().set_size_inches(10,4)

plt.savefig('figs/tuberculosis-patchevaluation.png', bbox_inches='tight')

y_pred = cnn.predict_proba(train_X)

false_positive_rate, true_positive_rate, thresholds = metrics.roc_curve(train_y, y_pred[:,1])
roc_auc = metrics.auc(false_positive_rate, true_positive_rate)

precision, recall, thresholds = metrics.precision_recall_curve(train_y, y_pred[:,1])
average_precision = metrics.average_precision_score(train_y, y_pred[:, 1])

subplot(121)
plt.title('ROC: AUC = %0.2f'% roc_auc)
plt.plot(false_positive_rate, true_positive_rate, 'b')
plt.legend(loc='lower right')
plt.plot([0,1],[0,1],'r--')
plt.ylim([-.05, 1.05])
plt.xlim([-.05, 1.0])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')

subplot(122)
plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('Precision-Recall: AP={0:0.2f}'.format(average_precision))
plt.legend(loc="lower left")

plt.gcf().set_size_inches(10,4)

plt.savefig('figs/tuberculosis-patchevaluation.png', bbox_inches='tight')

false_positive_rate, true_positive_rate, thresholds = metrics.roc_curve(test_y,...


I am working on the file "CNN.ipynb".
The first task is:
Also, calculate the class number which is the index of the largest element.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "CNN.ipynb".
The first task is:
Calculate the class number for each label in the test set.
Can you write Python code for it?

data.test.cls = np.argmax(data.test.labels, axis=1)


I am working on the file "CNN.ipynb".
The first task is:
Calculate the classification accuracy.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "CNN.ipynb".
The first task is:
Create a boolean vector where the prediction class equals the true class.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "CNN.ipynb".
The first task is:
Create a convolutional layer method in the computational graph for TensorFlow by using, input, number of channels, filter size, number of filters. Input is assumed as four-dimensional tensor with these (Image number, X-axis of image, Y-axis of image, Channels of each image) dimensions.
Can you write Python code for it?

def new_conv_layer(input, num_input_channels, filter_size, num_filters, use_pooling=True):

    shape = [filter_size, filter_size, num_input_channels, num_filters]
    weights = new_weights(shape=shape)
    biases = new_biases(length=num_filters)

    layer = tf.nn.conv2d(input=input,filter=weights,strides=[1, 1, 1, 1],padding='SAME')
    layer += biases

    if use_pooling:
        layer = tf.nn.max_pool(value=layer,ksize=[1, 2, 2, 1],strides=[1, 2, 2, 1],padding='SAME')

    layer = tf.nn.relu(layer)

    return layer, weights


I am working on the file "CNN.ipynb".
The first task is:
Create a new TensorFlow variables and initialize with the random values.
Can you write Python code for it?

def new_weights(shape):
    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))

def new_biases(length):
    return tf.Variable(tf.constant(0.05, shape=[length]))


I am working on the file "CNN.ipynb".
The first task is:
Create a second convolutional layer by again calling the convolutional layer method in which the output from the 1st convolutional layer is given as input to the second convolutional layer.
Can you write Python code for it?

layer_conv2, weights_conv2 = new_conv_layer(input=layer_conv1,num_input_channels=num_filters1,filter_size=filter_size2,
                                            num_filters=num_filters2, use_pooling=True)


I am working on the file "CNN.ipynb".
The first task is:
Create an optimize method to optimize the variables of the network layers and for each iteration, a new batch of data is selected from the training data.
Can you write Python code for it?

total_iterations = 0

def optimize(num_iterations):
    global total_iterations

    start_time = time.time()

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        feed_dict_train = {x: x_batch,y_true: y_true_batch}
        session.run(optimizer, feed_dict=feed_dict_train)

        if i % 100 == 0:
            acc = session.run(accuracy, feed_dict=feed_dict_train)
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"
            print(msg.format(i + 1, acc))

    total_iterations += num_iterations
    end_time = time.time()
    time_dif = end_time - start_time
    print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))


I am working on the file "CNN.ipynb".
The first task is:
Create method to plot images from the test set that are wrongly classified.
Can you write Python code for it?

def display_errors(cls_pred, correct):
    incorrect = (correct == False)
    images = data.test.images[incorrect]
    cls_pred = cls_pred[incorrect]
    cls_true = data.test.cls[incorrect]
    display_images(images=images[0:16],cls_true=cls_true[0:16],cls_pred=cls_pred[0:16])


I am working on the file "CNN.ipynb".
The first task is:
Define a placeholder variable for the input image.
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')


I am working on the file "CNN.ipynb".
The first task is:
Display the sizes of the labels in each sets of the MNIST data.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))


I am working on the file "CNN.ipynb".
The first task is:
Download and extract the training and testing images and labels sets from the MNIST dataset.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "CNN.ipynb".
The first task is:
Flatten the output of the 2nd convolutional layer from 4-dimensional tensor to 2-dimensional tensor.
Can you write Python code for it?

layer_flat, num_features = flatten_layer(layer_conv2)


I am working on the file "CNN.ipynb".
The first task is:
Initialize the filter size and number of filters for both the convolutional layers.
Also, intialize the total number of nodes in the convolutional layer.
Can you write Python code for it?

filter_size1 = 5
num_filters1 = 16

filter_size2 = 5
num_filters2 = 36

fc_size = 128


I am working on the file "CNN.ipynb".
The first task is:
Normalize the output of the second fully connected layer by using softmax method.
Can you write Python code for it?


I am working on the file "CNN.ipynb".
The first task is:
Now create a first convolutional layer by calling convolutional layer method as defined before and use pooling to down sample the image resolution by 2.
Can you write Python code for it?

layer_conv1, weights_conv1 = new_conv_layer(input=x_image,num_input_channels=num_channels,filter_size=filter_size1,
                                            num_filters=num_filters1,use_pooling=True)


I am working on the file "CNN.ipynb".
The first task is:
Now create a fully connected layer method with input of 2-dimension tensor and calculate the layers by calculating the mathematical formula weight*input + biases after creating weights and biases.
Can you write Python code for it?

def new_fc_layer(input,num_inputs,num_outputs,use_relu=True):

    weights = new_weights(shape=[num_inputs, num_outputs])
    biases = new_biases(length=num_outputs)

    layer = tf.matmul(input, weights) + biases

    if use_relu:
        layer = tf.nn.relu(layer)

    return layer


I am working on the file "CNN.ipynb".
The first task is:
Plot few sample images.
Can you write Python code for it?

images = data.test.images[0:16]
cls_true = data.test.cls[0:16]
display_images(images=images, cls_true=cls_true)


I am working on the file "CNN.ipynb".
The first task is:
Reshape the variable x to 4-dimension tensor because the input to the convolutional layer is 4-dimension tensor.
Can you write Python code for it?

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])


I am working on the file "CNN.ipynb".
The first task is:
Select certain batch of input images.
Can you write Python code for it?


I am working on the file "CNN.ipynb".
The first task is:
Similarly, create a placeholder variable for true labels.
Can you write Python code for it?

y_true = tf.placeholder(tf.float32, shape=[None, num_classes], name='y_true')


I am working on the file "CNN.ipynb".
The first task is:
Verify the dimension or shape of the tensor from the output of the first convolutional layer.
Can you write Python code for it?


I am working on the file "CNN/CNN_tf.ipynb".
The first task is:
CNN Feedforward test for Convolution Neural Networks
Can you write Python code for it?


I am working on the file "CNN/CNN_tf.ipynb".
The first task is:
CNN class test as a Deep Neural Network (i.e. Artificial neural network, i.e. no convolution)
Can you write Python code for it?

L=4
CorD=('D','D','D','D')
dims_data_test=[(784,392), (392,196),(196,98),(98,10)]

CNNFF_test=CNN_tf.Feedforward(L,CorD,dims_data_test,psi_L=tf.nn.softmax)

CNN_test=CNN_tf.CNN(CNNFF_test)

CNN_test.connect_through()

CNN_test.build_J_L2norm_w_reg(lambda_val=0.01)

CNN_test.build_optimizer()

CNN_test.train_model(max_iters=10000, X_data=train_set_x[:20000],y_data=y_train[:20000])

yhat_test = CNN_test.predict()

print(type(yhat_test))
print(yhat_test.shape)
print(np.argmax(yhat_test,axis=1).shape)
np.mean( train_set_y[:20000]==np.argmax(yhat_test,axis=1) )


I am working on the file "CNN/CNN_tf.ipynb".
The first task is:
Turn this into a so-called "one-hot vector representation."
Recall that whereas the original labels (in the variable y) were 0,1, ..., 9 for 10 different (single) digits, for the purpose of training a neural network, we need to recode these labels as vectors containing only values 0 or 1.
Can you write Python code for it?

K=10
m_train = train_set_y.shape[0]
m_valid = valid_set_y.shape[0]
m_test  = test_set_y.shape[0]
y_train = [np.zeros(K) for row in train_set_y] # list of m_train numpy arrays of size dims. (10,)
y_valid = [np.zeros(K) for row in valid_set_y] # list of m_valid numpy arrays of size dims. (10,)
y_test = [np.zeros(K) for row in test_set_y] # list of m_test numpy arrays of size dims. (10,)
for i in range(m_train):
    y_train[i][ int(train_set_y[i]) ] = 1.
for i in range(m_valid):
    y_valid[i][ int(valid_set_y[i]) ] = 1.
for i in range(m_test):
    y_test[i][ int(test_set_y[i]) ] = 1.
y_train = np.array(y_train).astype(np.float32)
y_valid = np.array(y_valid).astype(np.float32)
y_test = np.array(y_test).astype(np.float32)
print(y_train.shape)
print(y_valid.shape)
print(y_test.shape)

print(y_train.min())
print(y_train.max())
print(y_valid.min())
print(y_valid.max())
print(y_test.min())
print(y_test.max())


I am working on the file "CNN/DataDriven_StarNet_CNN.ipynb".
The first task is:
Build and compile model
Can you write Python code for it?

input_spec = Input(shape=(num_fluxes,1,))
model = build_model(input_spec)
optimizer = Adam(lr=lr, beta_1=beta_1, beta_2=beta_2, epsilon=optimizer_epsilon, decay=0.0)

early_stopping = EarlyStopping(monitor='val_loss', min_delta=early_stopping_min_delta,
                               patience=early_stopping_patience, verbose=2, mode='min')

reduce_lr = ReduceLROnPlateau(monitor='loss', factor=0.5, epsilon=reuce_lr_epsilon,
                              patience=reduce_lr_patience, min_lr=reduce_lr_min, mode='min', verbose=2)

model.compile(optimizer=optimizer, loss=loss_function)
model.summary()


I am working on the file "CNN/DataDriven_StarNet_CNN.ipynb".
The first task is:
Load non-normalized spectra
Can you write Python code for it?

# hack to load pre-computed mean and std-dev for faster normalization
mean_and_std = np.load('/data/stars/apogee/dr14/aspcap_labels_mean_and_std.npy')
mean_labels = mean_and_std[0]
std_labels = mean_and_std[1]
num_labels = mean_and_std.shape[1]

def normalize(lb):
    return (lb-mean_labels)/std_labels

data_file = '/data/stars/apogee/dr14/starnet_training_data.h5'

with h5py.File(data_file,"r") as F:
    spectra = F["spectrum"][:]
    labels = np.column_stack((F["TEFF"][:],F["LOGG"][:],F["FE_H"][:]))
    # Normalize labels
    labels = normalize(labels)
print('Reference set includes '+str(len(spectra))+' individual visit spectra.')

# define the number of wavelength bins (typically 7214)
num_fluxes = spectra.shape[1]
print('Each spectrum contains '+str(num_fluxes)+' wavelength bins')

num_train=int(0.9*len(labels))

# set NaN values to zero
indices_nan = np.where(np.isnan(spectra))
spectra[indices_nan]=0.

# some visit spectra are just zero-vectors... remove these.
spec_std = np.std(spectra,axis=1)
spec_std = spec_std.reshape(spec_std.shape[0],1)
indices = np.where(spec_std!=0.)[0]
spectra = spectra[indices]
labels = labels[indices]

reference_data = np.column_stack((spectra,labels))
np.random.shuffle(reference_data)

train_spectra = reference_data[0:num_train,0:num_fluxes]

# Reshape spectra for convolutional layers
train_spectra = train_spectra.reshape(train_spectra.shape[0], train_spectra.shape[1], 1)
train_labels = reference_data[0:num_train,num_fluxes:]

cv_spectra = reference_data[num_train:,0:num_fluxes]
cv_spectra = cv_spectra.reshape(cv_spectra.shape[0], cv_spectra.shape[1], 1)
cv_labels = reference_data[num_train:,num_fluxes:]

reference_data=[]
spectra=[]
labels=[]

print('Training set includes '+str(len(train_spectra))+' spectra and the cross-validation set includes '+str(len(cv_spectra))+' spectra')

time1 = time.time()

# Train model
model.fit(train_spectra, train_labels, validation_data=(cv_spectra, cv_labels),
          epochs=max_epochs, batch_size=batch_size, verbose=2,
         ...


I am working on the file "CNN/DataDriven_StarNet_CNN.ipynb".
The first task is:
Plot the input spectra, then the normalized spectra. I will force the second of the two plots to have the same y-axis range to ensure that the range for our normalized spectra are similar to one another
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

for i in range(10):
    fig, axes = plt.subplots(2,1,figsize=(70, 10))
    axes[0].plot(cv_spectra[i,:,0],c='b')
    axes[1].plot(normalized_cv[i,:,0],c='r')
    axes[1].set_ylim((-4,4))
    plt.show()


I am working on the file "CNN/DataDriven_StarNet_CNN.ipynb".
The first task is:
Spectra Normalization
Tentative to replace what stellar spectroscopist call normalization (supression of a global continuum over the whole spectrum) with an input normalization.
This test is simply a convolutional layer with one filter of length 1, followed by an InstanceNormalization layer
First build a model that only includes our input convolutional and instance normalization layers.
Note: I use a constant initialization of 0.5 because if the kernel is < 0. then the normalized spectra are inverted. this probably doesn't matter for the NN but it makes it a lot nicer to plot
Can you write Python code for it?

def build_normalizer_model(input_spec):

    # input conv layer with filter length 1 to flatten the shape
    x = Conv1D(kernel_initializer=keras.initializers.Constant(0.5), activation='linear', padding="same", filters=1,
           kernel_size=1,use_bias=False)(input_spec)
    # instance normalize to bring each spectrum to zero-mean and unit variance
    normed_spec = InstanceNormalization()(x)

    return Model(input_spec,normed_spec)

input_spec = Input(shape=(num_fluxes,1,))
model = build_normalizer_model(input_spec)
model.summary()
normalized_cv = model.predict(cv_spectra)


I am working on the file "CNN/DataDriven_StarNet_CNN.ipynb".
The first task is:
We may want to do some pre-processing clipping to the spectra to elminate the outliers
Stacking
Is the stacking method used on spectra to add them to the output from conv blocks correct?
First extend previous model to include the upsample layer.
Can you write Python code for it?

def build_upsample_model(input_spec):

    # input conv layer with filter length 1, no bias value
    x = Conv1D(kernel_initializer=keras.initializers.Constant(0.5), activation='linear', padding="same", filters=1,
           kernel_size=1,use_bias=False)(input_spec)
    # instance normalize to bring each spectrum to zero-mean and unit variance
    normed_spec = InstanceNormalization()(x)

    # upsample the spectra so that they can be easily added to the output of the conv layers
    # this method just repeats the spectra n=num_filters times
    normed_spec = Reshape((num_fluxes,1,1))(normed_spec)
    repeated_spec = UpSampling2D(size=(1, num_filters))(normed_spec)
    repeated_spec = Reshape((num_fluxes,num_filters))(repeated_spec)

    return Model(input_spec,repeated_spec)

input_spec = Input(shape=(num_fluxes,1,))
model = build_upsample_model(input_spec)
model.summary()
upsampled_cv = model.predict(cv_spectra[0:100])

# Plot the input spectra, then the normalized upsampled spectra
for i in range(5):
    fig, axes = plt.subplots(9,1,figsize=(70, 10))
    axes[0].plot(cv_spectra[i,:,0],c='b')
    for ii in range(8):
        axes[ii+1].plot(upsampled_cv[i,:,ii],c='r')
        axes[ii+1].set_ylim((-4,4))
    plt.show()


I am working on the file "CNN/fashion_mnist_cnn/fashion_cnn_tpu.ipynb".
The first task is:
Run the train phase
Can you write Python code for it?


I am working on the file "CNN/lenet_theano.ipynb".
The first task is:
CNN class test as a Deep Neural Network (i.e. Artificial neural network, i.e. no convolution)
Can you write Python code for it?

L=4
CorD=('D','D','D','D')
dims_data_test=[(784,392), (392,196),(196,98),(98,10)]

CNNFF_test=CNN.Feedforward(L,CorD,dims_data_test,psi_L=T.nnet.softmax)

CNN_test=CNN.CNN(CNNFF_test, X=train_set_x[:20000],y=y_train[:20000])

CNN_test.connect_through(X=train_set_x[:20000])
#CNN_test.connect_through()

CNN_test.build_J_L2norm_w_reg(0.001)

CNN_test.build_update(alpha=0.01,beta=0.000001)

CNN_test_train_errs=CNN_test.train_model_full(max_iters=20000)

result = theano.function([],CNN_test._CNN_model._get_outer_layer_())
result = result()

print(type(result))
print(result.shape)

train_set_y[:10]

np.argmax(result,axis=1)[:10]

np.mean( train_set_y[:20000] == np.argmax( result,axis=1) )


I am working on the file "CNN/lenet_theano.ipynb".
The first task is:
CNN class test for Convolution Neural Networks
Can you write Python code for it?

print(train_set_x.shape)

CNN_test = CNN.CNN(CNNFF_test,X=train_set_x[:20000],y=y_train[:20000])
#CNN_test = CNN.CNN(CNNFF_test,X=train_set_x,y=y_train)

#CNN_test.connect_through(X=train_set_x)
CNN_test.connect_through(X=train_set_x[:20000])

CNN_test.build_J_logistic_w_reg(0.001)

CNN_test.build_update(alpha=0.001, beta=0.000001)  # takes 1 minute, chill

CNN_test_train_errs = CNN_test.train_model_full(max_iters=20000) # 10:09 am for 200 < 1min., 20000 at 10:11 am-11:28 am

result = theano.function([],CNN_test._CNN_model._get_outer_layer_())
result = result()
print(type(result))
print(result.shape)
np.argmax(result,axis=1)[:10]

np.mean( train_set_y[:20000] == np.argmax( result,axis=1) )

# Save parameters like this:
CNN_test.save_parameters("../saved_models/CNN_MNIST.npy")

CNN_test.connect_through(X=valid_set_x)

CNN_test.y = theano.shared( y_valid.astype(theano.config.floatX),borrow=True)

result_valid = theano.function([],CNN_test._CNN_model._get_outer_layer_())
result_valid = result_valid()
print(type(result_valid))
print(result_valid.shape)
np.argmax(result_valid,axis=1)[:10]

np.mean( valid_set_y == np.argmax( result_valid,axis=1) )

CNN_test.connect_through(X=test_set_x)

result_test = theano.function([],CNN_test._CNN_model._get_outer_layer_())
result_test = result_test()
print(type(result_test))
print(result_test.shape)
np.argmax(result_test,axis=1)[:10]

np.mean( test_set_y == np.argmax( result_test,axis=1) )

#testXsh=theano.shared( np.array( range(1,12+1)).reshape(3,4).astype(theano.config.floatX))
#print(np.array( range(1,12+1)).reshape(3,4))

#testXsumshf = theano.function([],T.sum(testXsh,axis=1))
#testXsumshf_results = testXsumshf()

#testXsumshf_results

#12*13/2

#testaL4=theano.function([],CNN_test._CNN_model.Axons[3].al)
#testaL4=theano.function([],CNN_test._CNN_model._get_outer_layer_())

#testaL4().shape

#range(1,13)


I am working on the file "CNN/lenet_theano.ipynb".
The first task is:
Convolution axon test
Can you write Python code for it?

sys.path.append( "../ML/")

import CNN
from CNN import Axon_CNN

c_sizedims = (2,3,9,9) # (C_l,C_lm1,W_1,W_2) = (C_l,C_{l-1},W_1,W_2)
c_bound = np.sqrt(3*9*9)
c = theano.shared( np.asarray( rng.uniform(
                                        low=-1.0/c_bound,
                                        high=1.0/c_bound,
                                        size=c_sizedims),dtype=theano.config.floatX),name='c')

b_sizedim = (2,) # C_l=2
b = theano.shared( np.asarray(
                    rng.uniform(low=-.5, high=.5, size=b_sizedim), dtype=theano.config.floatX), name='b')

Conv_axon_test = Axon_CNN(0,(3,2),(9,9),Pl=None,c=c,b=b,activation=None)

Conv_axon_test.connect_through()

f_Conv_axon_test = theano.function([Conv_axon_test.alm1], Conv_axon_test.al)

filtered_img_Conv_axon_test = f_Conv_axon_test(img_)

# plot original image and first and second components of output
pylab.subplot(1, 3, 1); pylab.axis('off'); pylab.imshow(img)
pylab.gray();
# recall that the convOp output (filtered image) is actually a "minibatch",
# of size 1 here, so we take index 0 in the first dimension:
pylab.subplot(1, 3, 2); pylab.axis('off'); pylab.imshow(filtered_img_Conv_axon_test[0, 0, :, :])
pylab.subplot(1, 3, 3); pylab.axis('off'); pylab.imshow(filtered_img_Conv_axon_test[0, 1, :, :])
pylab.show()

print(type(img_))
print(img_.shape)
print(img_.max())
print(img_.min())
print(type(filtered_img_Conv_axon_test))
print(filtered_img_Conv_axon_test.shape)
print(filtered_img_Conv_axon_test.max())
print(filtered_img_Conv_axon_test.min())

"""
# for troubleshooting
l=0
C_ls=(3,2)
W_ls=(9,9)
C_lm1, C_l = C_ls
rng = np.random.RandomState(1234)
fan_in = C_lm1 * np.prod( W_ls)
fan_out = C_l * np.prod(W_ls) // np.prod((2,2))
filter_size = (C_l,C_lm1) + W_ls
"""

"""
# for troubleshooting
c_values = np.asarray( rng.uniform( low=-np.sqrt(6. / ( fan_in + fan_out )),
                                   high=np.sqrt(6. / ( fan_in + fan_out )), size=filter_size  ),
dtype=theano.config.floatX...


I am working on the file "CNN/lenet_theano.ipynb".
The first task is:
Empirically, we seen that the image "shrank" in size dimensions.  We can infer that this convolution operation doesn't assume anything about the boundary conditions, and so the filter (stencil), requiring a, in this case, 9x9 "block" or 9x9 values, will only, near the boundaries, output values for the "inside" cells/grid points.
MaxPooling
Can you write Python code for it?

input = T.dtensor4('input')
maxpool_shape=(2,2)
pool_out = pool.pool_2d(input, maxpool_shape, ignore_border=True)
f = theano.function([input],pool_out)

invals=np.random.RandomState(1).rand(3,2,5,5)
print(invals.shape)

f(invals).shape

pool_out_false = pool.pool_2d(input,maxpool_shape,ignore_border=False)
f_false = theano.function([input],pool_out_false)

f_false(invals).shape

invals2 = np.random.RandomState(1).rand(3,2,256,64)
print(f(invals2).shape)
print(f_false(invals2).shape)
invals3 = np.random.RandomState(1).rand(3,2,257,65)
print(f(invals3).shape)
print(f_false(invals3).shape)

maxpool_shape2=(4,4)
pool_out2 = pool.pool_2d(input, maxpool_shape2, ignore_border=True)
f2 = theano.function([input],pool_out2)
print(f2(invals2).shape)

C_ls_test = (4,6)
W_ls_test = (640,510)
C_ls_test+W_ls_test

type(C_ls_test) is type( (0,))

len(C_ls_test)

np.stack( (np.array(C_ls_test), np.array(W_ls_test)),axis=1)


I am working on the file "CNN/lenet_theano.ipynb".
The first task is:
Let's have a little bit of fun with this:
Can you write Python code for it?

import pylab
from PIL import Image

# open example image of dimensions 639x516, HxW
img = Image.open(open('../Data/3wolfmoon.jpg'))

print(img.size)  # WxH
print(np.asarray(img).max())
print(np.asarray(img).min())

# dimensions are (height,width,channel)
img_np=np.asarray(img,dtype=theano.config.floatX)/256.
print(img_np.shape)
print(img_np.max())
print(img_np.min())

# put image in 4D tensor of shape (1,3,height,width)
img_ = img_np.transpose(2,0,1).reshape(1,3,639,516)

filtered_img = f(img_)

# plot original image and first and second components of output
pylab.subplot(1, 3, 1); pylab.axis('off'); pylab.imshow(img)
pylab.gray();
# recall that the convOp output (filtered image) is actually a "minibatch",
# of size 1 here, so we take index 0 in the first dimension:
pylab.subplot(1, 3, 2); pylab.axis('off'); pylab.imshow(filtered_img[0, 0, :, :])
pylab.subplot(1, 3, 3); pylab.axis('off'); pylab.imshow(filtered_img[0, 1, :, :])
pylab.show()

print(type(img_))
print(img_.shape)
print(img_.max())
print(img_.min())
print(type(filtered_img))
print(filtered_img.shape)
print(filtered_img.max())
print(filtered_img.min())


I am working on the file "CNN/lenet_theano.ipynb".
The first task is:
MNIST Data
cf. Found here http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz or https://www-labs.iro.umontreal.ca/~lisa/deep/data/mnist/?C=M;O=A
Can you write Python code for it?

import os, sys
import gzip
import six.moves.cPickle as pickle

import numpy
import numpy as np

# find where `mnist.pkl.gz` is on your own computer
f=gzip.open("../Data/mnist.pkl.gz",'rb')
try:
    train_set,valid_set,test_set = pickle.load(f,encoding='latin1')
except:
    train_set,valid_set,test_set = pickle.load(f)
f.close()

train_set_x,train_set_y=train_set
valid_set_x,valid_set_y=valid_set
test_set_x,test_set_y=test_set
train_set_x = train_set_x.astype(theano.config.floatX)
train_set_y = train_set_y.astype(theano.config.floatX)
valid_set_x = valid_set_x.astype(theano.config.floatX)
valid_set_y = valid_set_y.astype(theano.config.floatX)
test_set_x = test_set_x.astype(theano.config.floatX)
test_set_y = test_set_y.astype(theano.config.floatX)
print(train_set_x.shape,train_set_y.shape) # observe the value for (m,d), number of training examples x number of features
print(valid_set_x.shape,valid_set_y.shape)
print(test_set_x.shape,test_set_y.shape)

# this is for reshaping the MNIST data into something for convolution neural networks
train_set_x = train_set_x.reshape((train_set_x.shape[0],1,28,28))
valid_set_x = valid_set_x.reshape((valid_set_x.shape[0],1,28,28))
test_set_x = test_set_x.reshape((test_set_x.shape[0],1,28,28))
print(train_set_x.shape)
print(valid_set_x.shape)
print(test_set_x.shape)

print(train_set_y.min())
print(train_set_y.max())
print(valid_set_y.min())
print(valid_set_y.max())
print(test_set_y.min())
print(test_set_y.max())


I am working on the file "CNN/lenet_theano.ipynb".
The first task is:
Notice how
Can you write Python code for it?


I am working on the file "CNN101/CNN101_Test.ipynb".
The first task is:
Create Graph
Can you write Python code for it?

def filter2Gray(weight):
    """
    Convert filters in CNN into a batch of not yet normalized (e.g ranged from 0 to 255)gray images
    :type weight: A Tensor of shape [filter_height, filter_width, in_channels, out_channels]
    :type name: str
    :rtype: A Tensor of shape [number_of_filters, height, width, 1]
    """
    trans=tf.transpose(a=weight,perm=[2,3,0,1])
    ret=tf.reshape(tensor=trans,shape=[-1,int(trans.shape[2]),int(trans.shape[3]),1])
    return ret
with tf.name_scope("CNN"):
    with tf.name_scope("Input"):
        X=tf.placeholder(dtype=tf.float32,shape=[None,784],name="X")
        Y=tf.placeholder(dtype=tf.float32,shape=[None,10],name="Y")
        NetIn=tf.reshape(tensor=X,shape=[-1,28,28,1],name="NetIn")
    with tf.name_scope("Conv1"):
        W1=tf.Variable(tf.truncated_normal([5,5,1,32]),name="Weight")
        b1=tf.Variable(tf.truncated_normal([32]),name="bias")
        L1Out=tf.nn.conv2d(input=NetIn,filter=W1,strides=[1,1,1,1],padding="SAME",name="conv")
        L1Out=tf.nn.relu(L1Out+b1,name="ReLu")
        L1Out=tf.nn.max_pool(value=L1Out,ksize=[1,2,2,1],strides=[1,2,2,1],padding="SAME",name="L1Out")
        #Need to check out what the param#0 and param#3 means in ksize.
    with tf.name_scope("Conv2"):
        W2=tf.Variable(tf.truncated_normal([5,5,32,64]),name="Weight")
        b2=tf.Variable(tf.truncated_normal([64]),name="bias")
        L2Out=tf.nn.conv2d(input=L1Out,filter=W2,strides=[1,1,1,1],padding="SAME",name="conv")
        L2Out=tf.nn.relu(L2Out+b2,name="ReLu")
        L2Out=tf.nn.max_pool(value=L2Out,ksize=[1,2,2,1],strides=[1,2,2,1],padding="SAME",name="L2Out")
        #Need to check out what the param#0 and param#3 means in ksize.
    with tf.name_scope("FullConnect"):
        flatten=tf.reshape(tensor=L2Out,shape=[-1,7*7*64],name="flatten")
        W3=tf.Variable(tf.truncated_normal(dtype=tf.float32,shape=[7*7*64,1024]),name="Weight")
        b3=tf.Variable(tf.truncated_normal(dtype=tf.float32,shape=[1024]),name="bias")
       ...


I am working on the file "CNN101/CNN101_Test.ipynb".
The first task is:
Load MNIST dataset
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
MNIST=input_data.read_data_sets(r"..\DataSet\mnist",one_hot=True)
print("Number of training samples: {}\nNumber of test samples: {}".format(MNIST.train.num_examples,MNIST.test.num_examples))


I am working on the file "CNN101/CNN101_Test.ipynb".
The first task is:
Reload Parameters and test performance
Can you write Python code for it?

batchSize=20
with tf.Session() as sess:
    saver=tf.train.Saver()
    saver.restore(sess=sess,save_path=r".\model_checkpoints\MNIST_CNN-"+str(3000))
    acc=0
    for batch_i in range(int(MNIST.test.num_examples/batchSize)):
        x_batch,y_batch=MNIST.test.next_batch(batch_size=batchSize)
        pred=sess.run(L4Out,feed_dict={X:x_batch})
        acc+=sess.run(tf.reduce_sum(tf.cast(x=tf.equal(tf.argmax(input=pred,axis=1),tf.argmax(input=y_batch,axis=1)),dtype=tf.float32)))
print("Accuracy: {}".format(acc/MNIST.test.num_examples))


I am working on the file "CNN101/CNN101_Training.ipynb".
The first task is:
Create Graph
Can you write Python code for it?


I am working on the file "CNN101/CNN101_Training.ipynb".
The first task is:
Load MNIST dataset
Can you write Python code for it?


I am working on the file "CNN101/CNN101_Training.ipynb".
The first task is:
Training
Can you write Python code for it?

batchSize=50
epochs_n=3000
with tf.Session() as sess:
    writer=tf.summary.FileWriter(logdir=r".\graphs",graph=sess.graph)
    saver=tf.train.Saver()
    sess.run(tf.global_variables_initializer())
    for epoch in range(1,epochs_n+1):
        x_batch,y_batch=MNIST.train.next_batch(batch_size=batchSize)
        _,summary=sess.run([optimizer,summary_op_epoch],feed_dict={X:x_batch,Y:y_batch})
        writer.add_summary(summary=summary,global_step=epoch)
        if (epoch-1)%10==0:
            saver.save(sess=sess,save_path=r".\model_checkpoints\MNIST_CNN",global_step=epochs_n)
    summary=sess.run(summary_op_final)
    writer.add_summary(summary=summary,global_step=epochs_n)
    saver.save(sess=sess,save_path=r".\model_checkpoints\MNIST_CNN",global_step=epochs_n)
    writer.close()


I am working on the file "CNNs in TensorFlow.ipynb".
The first task is:
All preprocessing steps can be found in [ xyz .py ].
Building the TensorFlow graph
At the beginnig I promised a final accuracy of ~99.1%. For this we are going to use a network with 10 Layers and a ResNet-like extension. But be !! WARNED !! training that one on a Nvidia GTX 1050Ti takes around 50 hours. The final network can be found in [ xyz .py ].
For now we will just go with the general structure:
| In |                      RGB 48x48                     |
|----|:--------------------------------------------------:|
| 1  | Conv 5x5, 96 / ReLU -> Local Resp. Norm. -> Max Pool 2x2 |
| 2  | Conv 3x3, 96 / ReLU -> Local Resp. Norm. -> Max Pool 2x2 |
| 3  |                 Conv 3x3, 48 / ReLU                |
| 4  |                 Conv 3x3, 48 / ReLU                |
| 5  | Conv 3x3, 48 / ReLU  -> Local Resp. Norm. -> Max Pool 2x2 |
| 6  | Conv 2x2, 48 / ReLU -> Local Resp. Norm. -> Max Pool 2x2 |
| 7  |          FC Layer 3+6, 2048 / ReLU -> Dropout         |
| 8  |           FC Layer 6, 1024 / ReLU -> Dropout          |
| 9  |          FC Layer 7+8, 768 / ReLU -> Dropout          |
| 10 |                        FC 43                       |
Since it's kinda difficult to find code examples for working with the images-tab in TensorBoard I added a few nice visualizations
Can you write Python code for it?

import skimage

def variable_summaries(var, summary_name):
	with tf.name_scope("Summary_" + str(summary_name)):
		tf.summary.scalar("raw", var)
		tf.summary.histogram('histogram', var)

def convertToUINT8(x):
	return skimage.img_as_ubyte(x / np.amax(x))

## Based on panmari's post. Thank you!
## https://gist.github.com/panmari/4622b78ce21e44e2d69c

def VisualizeConvolutions(myTensor, sizeInfo,  name):
	V = tf.slice(myTensor, (0, 0, 0, 0), (1, -1, -1, -1), name='slice_' + name)
	V = tf.reshape(V, (sizeInfo[0], sizeInfo[1], sizeInfo[2]))
	V = tf.transpose(V, (2, 0, 1))
	V = tf.reshape(V, (-1, sizeInfo[0], sizeInfo[1], 1))
	tf.summary.image(name, V,  max_outputs=3)

## Add this in your graph definition to use TensorBoard's "IMAGE" Tab
## This will print the first 32 feature maps of each layer after the convolution
## as well as the original input image

with tf.name_scope('Visualiz_convolutions'):
    img_raw = tf.slice(x_image, (0, 0, 0, 0), (1, -1, -1, -1), name='slice_raw_image')
    tf.summary.image("raw_image", img_raw,  max_outputs=32)

    ## note: (width, height, output kernels)
    ## 48px = orig. image // 24px = feature map after maxpool
    VisualizeConvolutions(h_conv1, [48, 48, 96], "Conv1")
    VisualizeConvolutions(h_conv2, [24, 24, 96], "Conv2")
    VisualizeConvolutions(h_conv3, [12, 12, 48], "Conv3")


I am working on the file "CNNs in TensorFlow.ipynb".
The first task is:
To acutally see something you need to add some lines while running your session:
Can you write Python code for it?

logs_path = r'C:/....'
cross_entropy = (...)
accuracy = (...)

variable_summaries(cross_entropy, 'crossentropy')
variable_summaries(accuracy, 'TrainingAcc')
summary_op = tf.summary.merge_all()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    summary_writer = tf.summary.FileWriter(logs_path)
    summary_writer.add_graph(sess.graph)

    #Start training and logging
    for i in range(100):
        (...)
        _, summary = sess.run([train_step, summary_op], feed_dict={(...)})
        summary_writer.add_summary(summary, i)


I am working on the file "CNNs in TensorFlow.ipynb".
The first task is:
With this we have expanded our dataset by factor 8. Let's look at the distribution of datapoints over all classes:
<img src="files/Resources/Distribution_enh_skewed.png">
Working with a skewed dataset like this isn't a big problem. But since we want to be sure not to learn the distribution we are going to split our set in a skewed and an equalized set with 200 random samples per class. We will use the equalized set for a second training.
Can you write Python code for it?

import random

## our expanded dataset is stored in
enh_images, enh_labels = (...)

#since we need to know the start and end of the current class
lower_bound = 0
upper_bound = len(enh_images)

eq_images = []
eq_labels = []

#assumtion: class indizes are sorted and increasing
for classindex in range(43):
	for position in range(lower_bound, upper_bound):
		if enh_labels[position] != classindex:
			upper_bound	= position
			break
	sample_index= random.sample(range(lower_bound, upper_bound), 200)

	for i in sample_index:
		eq_images.append(enh_images[i][:])

	for i in sample_index:
		eq_labels.append(enh_labels[i])

	cnt = 0
	for i in sample_index:
		enh_images.pop(i - cnt)
		enh_labels.pop(i - cnt)
		cnt +=1

	lower_bound = upper_bound
	upper_bound = len(enh_images)


I am working on the file "CNNs/ECG-CNN-2D-VCG.ipynb".
The first task is:
Import and process data
Can you write Python code for it?

def importData(filepath):
    ppt = np.genfromtxt(filepath)
    dppt = np.diff(np.transpose(ppt))
    print(filepath, "Shape:", dppt[1:16,:].shape)
    return dppt[1:16,:]

pathIll = "./inData/clean_ecg/ill/"
pathHealth = "./inData/clean_ecg/health/"
illLst = []
healthLst = []

for file in os.listdir(pathIll):
    illLst.append(importData(pathIll+file))
for file in os.listdir(pathHealth):
    healthLst.append(importData(pathHealth+file))

print("Outputing Frank leads")
healthPat = np.concatenate((healthLst[:]), axis=1)[12:15]
illPat = np.concatenate((illLst[:]), axis=1)[12:15]

print(healthPat.shape, illPat.shape)

def findAbove(arr, threshold, skip):
    """
    Return indices for values above threshhold in array, arr. Keep only first items in sequence.
    """
    inlst = []
    for index, item in enumerate(arr):
        if item >= threshold:
            inlst.append(index)

    return inlst[::skip]

def processClassData(classData):
    """
    Process classData.

    Returns a one-hot array of shape [len(classData), 2].
    """
    # Convert label data to one-hot array

    classDataOH = np.zeros((len(classData),2))
    classDataOH[np.arange(len(classData)), classData] = 1

    return classDataOH

def getSamples(Arr, indexArr, buffer):
    """
    Get samples for inputting into CNN.
    """
    sampleArr = []

    for index, item in enumerate(indexArr):
        if Arr[0:, item-buffer:item+buffer].shape != (Arr.shape[0], buffer*2):
            pass
        else:
            sampleArr.append(Arr[0:, item-buffer:item+buffer])

    return np.array(sampleArr)

def visualiseData(ecgData, classData, gridSize, axis):
    """
    Plot labelled example data in a gridSize*gridSize grid.
    """
    fig, ax = plt.subplots(gridSize, gridSize, subplot_kw=dict(projection='3d'))
    plt.suptitle("Labelled example data")
    r = randint(0,len(classData)-ecgData.shape[1])
    k = 0
    if gridSize == 1:
        ax.plot(ecgData[r+k,0], ecgData[r+k,1], ecgData[r+k,2])
    else:
        for i...


I am working on the file "CNNs/ECG-CNN-2D-VCG.ipynb".
The first task is:
Neural Network
Can you write Python code for it?

sess = tf.InteractiveSession()
tf.reset_default_graph()
tflearn.initializations.normal()

# ecgData = np.zeros((50,12,400,1)) # If ecgData is not defined
# Input layer:
net = tflearn.layers.core.input_data(shape=[None, buffer*2, buffer*2, buffer*2, 1])

# First layer:
net = tflearn.layers.conv.conv_3d(net, 32, 5, activation="leaky_relu")
net = tflearn.layers.conv.max_pool_3d(net, 2)

# Second layer:
net = tflearn.layers.conv.conv_3d(net, 64, 5, activation="leaky_relu")
net = tflearn.layers.conv.max_pool_3d(net, 2)
net = tflearn.layers.core.flatten(net)

# Fully connected layer 1:
net = tflearn.layers.core.fully_connected(net, 1024, regularizer="L2", weight_decay=0.001, activation="leaky_relu")

# Dropout layer:
net = tflearn.layers.core.dropout(net, keep_prob=0.5)

# Output layer:
net = tflearn.layers.core.fully_connected(net, 2, activation="softmax")

net = tflearn.layers.estimator.regression(net, optimizer='adam', loss='categorical_crossentropy',\
                         learning_rate=0.00001)

model = tflearn.DNN(net, tensorboard_verbose=3)
model.fit(trainData, trainLabels, n_epoch=1, show_metric=True)

# Save model?
#now = datetime.datetime.now()
#model.save("./outData/models/cleanECG_2dconv_12lead_"+now.isoformat()+"_.tflearn")


I am working on the file "CNNs/ECG-CNN-2D-VCG.ipynb".
The first task is:
What if the model hasn't seen data from the patient? What then?!
Can you write Python code for it?

tpathIll = "./inData/clean_ecg/testIll/"
tpathHealth = "./inData/clean_ecg/testHealth/"
tillLst = []
thealthLst = []

for file in os.listdir(tpathIll):
    tillLst.append(importData(tpathIll+file))
for file in os.listdir(tpathHealth):
    thealthLst.append(importData(tpathHealth+file))

if frank == False:
    print("Outputing standard ECG leads...")
    thealth = np.concatenate((thealthLst[:]), axis=1)[0:12]
    till = np.concatenate((tillLst[:]), axis=1)[0:12]

elif frank == True:
    print("Outputing Frank leads...")
    thealth = np.concatenate((thealthLst[:]), axis=1)[12:15]
    till = np.concatenate((tillLst[:]), axis=1)[12:15]

print(thealth.shape, till.shape)

unseenData, unseenClassOH, unseenClass = functionTownCat(till, thealth, illThreshold, healthThreshold, \
                                                   skip, shift, buffer, True)

# Undifferentiate values
unseenData = undiff(unseenData, buffer)
tillarr, thealtharr = splitData(unseenData, unseenClass)

sens = model.evaluate(np.array(thealtharr), np.tile([1,0], (len(thealtharr), 1)))[0]
spec = model.evaluate(np.array(tillarr), np.tile([0,1], (len(tillarr), 1)))[0]
acc = model.evaluate(unseenData, unseenClassOH)[0]
lenh = len(thealtharr)
leni = len(tillarr)

print("Sensitivity:", sens,\
      "Specifity:", spec,\
      "Accuracy:", acc)

visualiseData(np.reshape(unseenData,(-1,unseenData.shape[1],buffer*2))[:,:,::20], unseenClass, 3, False)


I am working on the file "CNNs/ECG-CNN-2D.ipynb".
The first task is:
Import and process data
Can you write Python code for it?


I am working on the file "CNNs/ECG-CNN-2D.ipynb".
The first task is:
Neural Network
Can you write Python code for it?


I am working on the file "CNNs/ECG-CNN-2D.ipynb".
The first task is:
Test accuracy of model(s)
20% of training data held back for testing (4000 "heartbeats")
Can you write Python code for it?

#model.load("./outData/models/cleanECG_undiff_20e_300buff_0shift_2017-02-21T19:20:35.702943_.tflearn")
#model.load("./outData/models/cleanECG_undiff_20e_150buff_2017-02-21T16:15:02.602923_.tflearn")
#model.load("./outData/models/cleanECG_2dconv_12lead_2017-03-08T10:15:17.200943_.tflearn")
#model.load("./outData/models/cleanECG_2dconv_12lead_2017-03-09T18:05:18.655939_.tflearn")

labellst = classData[:round(ecgData.shape[0]*0.2)]
healthTest = []
illTest = []
for index, item in enumerate(labellst):
    if item == 1:
        illTest.append(testData[index])
    if item == 0:
        healthTest.append(testData[index])

healthLabel = np.tile([1,0], (len(healthTest), 1))
illLabel = np.tile([0,1], (len(illTest), 1))

print("Sensitivity:", model.evaluate(np.array(healthTest), healthLabel), "Specifity:",\
    model.evaluate(np.array(illTest), illLabel),\
    "Accuracy:", model.evaluate(testData, testLabels))


I am working on the file "CNNs/ECG-CNN-2D.ipynb".
The first task is:
What if the model hasn't seen data from the patient? What then?!
Can you write Python code for it?


I am working on the file "CNNs/MCG-ProcessData-3D.ipynb".
The first task is:
Make a visualisation of the data
Can you write Python code for it?

ecgData = np.load("./inData/3D-conv/ecgData.npy")
unseenData = np.load("./inData/3D-conv/unseenData.npy")

fig, ax = plt.subplots(3,3)
k = 200
fig.suptitle("Still frames of 3D MCG data")
for i in np.arange(3):
    for j in np.arange(3):
        ax[i,j].set_title("%s ms"%(k/2), fontsize=7)
        ax[i,j].imshow(ecgData[100,k], cmap="gray")
        ax[i,j].axis("off")
        k = k + 200

plt.savefig("/tmp/3Dmcg_frames.pdf")

ppt=100
fig = plt.figure()
data = ecgData[ppt,0]
im = plt.imshow(data)

def animate(i):
    data = ecgData[ppt,i]
    im.set_data(data)
    return im

plt.axis("off")
plt.title("Example of 3D (two space, \n 1 time dimension) MCG data used")
anim = animation.FuncAnimation(fig, animate, frames=np.arange(2000)[::10], repeat=False)
anim.save("/tmp/3D-Data-Example.mp4")


I am working on the file "CNNs/MCG-ProcessData-3D.ipynb".
The first task is:
This ipynb processes the data for use in the neural net found at MCG-CNN-3D.ipynb
Can you write Python code for it?

k = 1 # How many folds in the k-fold x-validation

## I used this to save the array in a smaller file so it doesn't eat all my ram
# df60 = pd.read_pickle("./inData/6060DF_MFMts.pkl")
# coilData = df60["MFMts"].as_matrix()
# ziData = np.zeros([400,2000,19,17])
#
# for i in np.arange(400):
#    for j in np.arange(2000):
#        ziData[i,j] = np.array(coilData[i][j][2])
#
# np.save("./inData/ziData.dat", ziData)
#
# ziClass = df60["Classification"].as_matrix()
# np.save("./inData/ziClass.dat", ziClass)

def splitData(coilData, classData):
    """
    Split data into healthy and ill types.
    """
    illData = []
    healthData = []

    for index, item in enumerate(classData):
        if item == 1:
            illData.append(coilData[index])
        if item == 0:
            healthData.append(coilData[index])

    return illData, healthData

classData = np.load("./inData/ziClass.npy")
coilData = np.load("./inData/ziData.npy")

# Normalise coilData for each unit time
for i in np.arange(coilData.shape[0]):
    for j in np.arange(coilData.shape[1]):
        coilData[i,j] = normalize(coilData[i,j], axis=1)

illData, healthData = splitData(coilData, classData)
if k == 1:
    illUnseen = np.array(illData[:20])
    healthUnseen = np.array(healthData[:20])
    illData = np.array(illData[20:])
    healthData = np.array(healthData[20:])
    print(illData.shape, healthData.shape,"\n", illUnseen.shape, healthUnseen.shape)
else:
    illData = np.array(illData)
    healthData = np.array(healthData)
    print(illData.shape, healthData.shape)

def processClassData(classData):
    """
    Process classData.

    Returns a one-hot array of shape [len(classData), 2].
    """
    # Convert label data to one-hot array

    classDataOH = np.zeros((len(classData),2))
    classDataOH[np.arange(len(classData)), classData] = 1

    return classDataOH

def functionTown(illArr, healthArr, shuffle):
    """
    Return the processed ecgData and the classData (one-hot). Also return arrays of ill and...


I am working on the file "CNTK_model_scoring_on_Spark_walkthrough.ipynb".
The first task is:

Evaluate model accuracy
The trained model assigns a class label (represented by an integer value 0-9) to each image. We now compare the true and predicted class labels to evaluate our model's accuracy.
Can you write Python code for it?

df = pd.DataFrame(results, columns=['true_label', 'predicted_label'])

num_correct = sum(df['true_label'] == df['predicted_label'])
num_total = len(results)
print('Correctly predicted {} of {} images ({:0.2f}%)'.format(num_correct, num_total, 100 * num_correct / num_total))


I am working on the file "CNTK_model_scoring_on_Spark_walkthrough.ipynb".
The first task is:

Score the images on worker nodes
The code cell below maps each partition of image_rdd to a worker node and collects the results. Runtimes of 1-3 minutes are typical.
Can you write Python code for it?

labelled_images = image_rdd.mapPartitions(run_worker)

# Time how long it takes to score 10k test images
start = pd.datetime.now()
results = labelled_images.collect()
print('Scored {} images'.format(len(results)))
stop = pd.datetime.now()
print(stop - start)


I am working on the file "CNTK_model_scoring_on_Spark_walkthrough.ipynb".
The first task is:

Convert the dataset into an RDD
The following code cell illustrates how the collection of images can be distributed to create a Spark RDD. The cell creates an RDD with one partition per worker to limit the number of times that the trained model must be reloaded during scoring.
Can you write Python code for it?

def reshape_image(record):
    image, label, filename = record
    return image.reshape(3,32,32).transpose(1,2,0), label, filename

image_rdd = sc.parallelize(zip(test_dict['data'], test_dict['labels'], test_dict['filenames']))
image_rdd = image_rdd.map(reshape_image)


I am working on the file "CNTK_model_scoring_on_Spark_walkthrough.ipynb".
The first task is:

Score the images using a trained CNTK model
Now that the cluster and sample dataset have been created, we can use PySpark to apply a trained model to the images.

Download the trained CNTK model and mean image to the Spark cluster
We previously trained a twenty-layer ResNet model to classify CIFAR-10 images by following this tutorial from the CNTK git repo. The model expects input images to be preprocessed by subtracting the mean image defined in an OpenCV XML file. The following cell downloads both the trained model and the mean image, and ensures that data from both files can be accessed by worker nodes.
Can you write Python code for it?

urlretrieve(model_uri, local_model_path)
sc.addFile(local_model_path)

urlretrieve(mean_image_uri, local_mean_image_path)
mean_image = xml.etree.ElementTree.parse(local_mean_image_path).getroot()
mean_image = [float(i) for i in mean_image.find('MeanImg').find('data').text.strip().split(' ')]
mean_image = np.array(mean_image).reshape((32, 32, 3)).transpose((2, 0, 1))
mean_image_bc = sc.broadcast(mean_image)


I am working on the file "CNTK_model_scoring_on_Spark_walkthrough.ipynb".
The first task is:

Download the dataset locally on the Spark cluster
The image data are ndarrays stored in a Python dict which has been pickled and tarballed. The cell below downloads the tarball and extracts the dict containing the test image data.
Can you write Python code for it?

if not os.path.exists(local_cifar_path):
    urlretrieve(cifar_uri, filename=local_cifar_path)

with tarfile.open(local_cifar_path, 'r:gz') as f:
    test_dict = pickle.load(f.extractfile('cifar-10-batches-py/test_batch'), encoding='latin1')


I am working on the file "CNTK_model_scoring_on_Spark_walkthrough.ipynb".
The first task is:
We can construct a confusion matrix to visualize which classification errors are most common:
Can you write Python code for it?

spark.createDataFrame(df).coalesce(1).write.mode("overwrite").csv("/tmp/cifar_scores", header=True)

import pandas as pd
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
import os
from glob import glob

df = pd.read_csv(glob('/tmp/cifar_scores/*.csv')[0])
print('Constructing a confusion matrix with the first {} samples'.format(len(df.index)))

label_to_name_dict = {0: 'airplane',
                      1: 'automobile',
                      2: 'bird',
                      3: 'cat',
                      4: 'deer',
                      5: 'dog',
                      6: 'frog',
                      7: 'horse',
                      8: 'ship',
                      9: 'truck'}

labels = np.sort(df['true_label'].unique())
named_labels = [label_to_name_dict[i] for i in labels]
cm = confusion_matrix(df['true_label'], df['predicted_label'], labels=labels)

plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
plt.colorbar()
tick_marks = np.arange(len(labels))
plt.xticks(tick_marks, named_labels, rotation=90)
plt.yticks(tick_marks, named_labels)
plt.xlabel('Predicted label')
plt.ylabel('True Label')
plt.show()


I am working on the file "CNV_dangerosite.ipynb".
The first task is:
Create a dict of the cnv
Can you write Python code for it?

from collections import OrderedDict
cnv = OrderedDict()
names_from = ["CHR de Merge_CIA_610_660_QC", 'START', 'STOP']
                        #, "5'gene", "3'gene", "5'dist(kb)", "3'dist(kb)"]
blank_dgr = 0

for line in arr[1:]:
    lline = line.split('\t')
    dgr = lline[i_DANGER]
    scr  = lline[i_SCORE]
    cnv_iid = util.make_uiid(line, names_from, arr[0])

    if dgr != '':
        add_cnv = (util.danger_score(lline[i_DANGER], util.pH1_with_apriori),
                   score2prob(util.process_one_score(lline[i_SCORE], max_score)))
        if cnv_iid in cnv.keys():
            cnv[cnv_iid].append(add_cnv)
        else:
            cnv[cnv_iid] = [add_cnv]
    else:
        blank_dgr += 1

print(len(cnv), (blank_dgr))
print([k for k in cnv.keys()[:5]])
print([k for k in cnv.values()[:5]])

for k in cnv.keys()[3340:3350]:
    print(k,': ',cnv[k])


I am working on the file "CNV_dangerosite.ipynb".
The first task is:
Create a dictionary of the subjects -
Can you write Python code for it?

cnv = OrderedDict({})
#names_from = ['START', 'STOP', "5'gene", "3'gene", "5'dist(kb)", "3'dist(kb)"]
names_from = ['IID_projet']

for line in arr[1:]:
    lline = line.split('\t')
    dgr = lline[i_DANGER]
    scr  = lline[i_SCORE]
    sub_iid = util.make_uiid(line, names_from, arr[0])

    if dgr != '':
        add_cnv = (util.danger_score(lline[i_DANGER], util.pH1_with_apriori),
                   score2prob(util.process_one_score(lline[i_SCORE], max_score)))
        if sub_iid in cnv.keys():
            cnv[sub_iid].append(add_cnv)
        else:
            cnv[sub_iid] = [add_cnv]


I am working on the file "CNV_dangerosite.ipynb".
The first task is:
Finally, putting things together
Can you write Python code for it?

# re-loading
reload(util)
CWD = osp.join(osp.expanduser('~'), 'documents','grants_projects','roberto_projects', \
               'guillaume_huguet_CNV','File_OK')
filename = 'Imagen_QC_CIA_MMAP_V2_Annotation.tsv'
fullfname = osp.join(CWD, filename)

# in numpy array
arr = np.loadtxt(fullfname, dtype='str', comments=None, delimiter='\Tab',
                 converters=None, skiprows=0, usecols=None, unpack=False, ndmin=0)

line0 = arr[0].split('\t')

i_DANGER = line0.index('Pvalue_MMAP_V2_sans_intron_and_Intergenic')
i_SCORE = line0.index('SCORE')
i_START = line0.index('START')
i_STOP = line0.index('STOP')
i_5pGENE = line0.index("5'gene")
i_3pGENE = line0.index("3'gene")
i_5pDIST = line0.index("5'dist(kb)")
i_3pDIST = line0.index("3'dist(kb)")
#i_LOC = line0.index('Location')

scores = np.asarray([line.split('\t')[i_SCORE] for line in arr[1:]])
clean_score = util.process_scores(scores)
max_score = clean_score.max()

print(line0)

#names_from = ['START', 'STOP', "5'gene", "3'gene", "5'dist(kb)", "3'dist(kb)"]

#---------- ligne uniques:
names_from = ['IID_projet', 'IID_genotype', "CHR de Merge_CIA_610_660_QC", 'START', 'STOP']
cnv_names = util.make_uiid(arr, names_from)
print("with names from: ", names_from)
print("we have {} unique elements out of {} rows in the tsv".format(
                                len(np.unique(cnv_names)), len(cnv_names)))

#---------- CNV uniques ?
names_from = ["CHR de Merge_CIA_610_660_QC", 'START', 'STOP']
cnv_names = util.make_uiid(arr, names_from)
print("with names from: ", names_from)
print("we have {} unique elements out of {} rows in the tsv".format(
                                len(np.unique(cnv_names)), len(cnv_names)))

#---------- sujets uniques ?
names_from = ['IID_projet'] # , 'IID_genotype']
cnv_names = util.make_uiid(arr, names_from)
print("with names from: ", names_from)
print("we have {} unique elements out of {} rows in the tsv".format(
                                len(np.unique(cnv_names)), len(cnv_names)))

dangers =...


I am working on the file "CNV_dangerosite.ipynb".
The first task is:
Histogram of the number of cnv used to compute dangerosity
Can you write Python code for it?

print(len(cnv))
nbcnv = [len(cnv[sb]) for sb in cnv]
hist = plt.hist(nbcnv, bins=50)
print(np.max(np.asarray(nbcnv)))

# definition of dangerosity from a list of cnv
def dangerosity(listofcnvs):
    """
    inputs: list tuples (danger_score, proba_cnv)
    returns: a dangerosity score
    """
    last = -1 #slicing the last
    tmp = [np.asarray(t) for t in zip(*listofcnvs)]
    return tmp[0].dot(tmp[1])

# or: return np.asarray([dgr*prob for (dgr,prob) in listofcnvs]).cumsum()[last]


I am working on the file "CNV_dangerosite.ipynb".
The first task is:
Printing out results
Can you write Python code for it?

dtime = datetime.now().strftime("%y-%m-%d_h%H-%M")
outfile = dtime+'dangerosity_cnv.txt'
fulloutfile = osp.join(CWD, outfile)

with open(fulloutfile, 'w') as outf:
    for sub in cnv:
        outf.write("\t".join([sub, str(dangerosity(cnv[sub]))]) + "\n")


I am working on the file "CNV_dangerosite.ipynb".
The first task is:
QUESTION pour Guillaume:
a quoi correspondent les '' dans la colonne "Pvalue_MMAP_V2_sans_intron_and_Intergenic" (danger)?
Ansewer: cnv for which we have no dangerosity information
Can you write Python code for it?

"""
danger_not_empty = dangers != ''
danger_scores = dangers[danger_not_empty]
danger_scores = np.asarray([util.danger_score(pstr, util.pH1_with_apriori)
                                                    for pstr in danger_scores])
""";


I am working on the file "CNV_dangerosite.ipynb".
The first task is:
To be or not to be a CNV: p value from the 'SCORE' column
Can you write Python code for it?

reload(util)
#get the scores
scores = np.asarray([line.split('\t')[i_score] for line in arr[1:]])
assert len(scores) == expected_nb_values
print(len(np.unique(scores)))
#tmp_score = np.asarray([util.str2floats(s, comma2point=True, sep=' ')[0] for s in scores])
assert scores.shape[0] == EXPECTED_LINES - 1

# h = plt.hist(tmp[tmp > sst.scoreatpercentile(tmp, 99)], bins=100)
# h = plt.hist(tmp[tmp < 50], bins=100)

"""
print("# CNV with score == 0.: ", (tmp==0.).sum())
print("# CNV with score >=15 < 17.5 : ", np.logical_and(tmp >= 15., tmp < 17.5).sum())
tmp.max()
""";


I am working on the file "CNV_dangerosite.ipynb".
The first task is:
Transforms the scores into P(cnv is real)
Can you write Python code for it?

# Creating a function from score to proba from Guillaume's values
p_cnv = util._build_dict_prob_cnv()
#print(p_cnv.keys())
#print(p_cnv.values())

#### Definition with a piecewise linear function
#score2prob = util.create_score2prob_lin_piecewise(p_cnv)
#scores = np.arange(15,50,1)
#probs = [score2prob(sc) for sc in scores]
#plt.plot(scores, probs)

#### Definition with a corrected regression line
score2prob = util.create_score2prob_lin(p_cnv)
#x = np.arange(0,50,1)
#plt.plot(x, [score2prob(_) for _ in x], '-', p_cnv.keys(), p_cnv.values(), '+')

p_scores = [score2prob(sc) for sc in clean_score]
assert len(p_scores) == EXPECTED_LINES -1


I am working on the file "CNV_dangerosite.ipynb".
The first task is:
transforming the "Pvalue_MMAP_V2_..." into danger score
Testing the function danger_score
Can you write Python code for it?

assert  util._test_danger_score_1()
assert  util._test_danger_score()


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
Create a dict of the cnv
Can you write Python code for it?

#gene_name_set = set([line.split('\t')[i_3pGENE] for line in arr[1:]]) | \
#           set([line.split('\t')[i_5pGENE] for line in arr[1:]])
#print('len(gene_name_set): ', len(gene_name_set))

# this creates an dict where the keys are unique indentifiers of cnv, values
# are (dangerosity, probability_cnv)

cnv = OrderedDict({})
names_from = ["CHR de Merge_CIA_610_660_QC", 'START', 'STOP']  #, "5'gene", "3'gene", "5'dist(kb)", "3'dist(kb)"]
blank_dgr = 0

for line in arr[1:]:
    lline = line.split('\t')
    dgr = lline[i_DANGER]
    scr  = lline[i_SCORE]
    cnv_iid = cnv_uiid(line, names_from, arr[0])

    if dgr != '':
        add_cnv = (danger_score(lline[i_DANGER], pH1_with_apriori),
                   score2prob(lline[i_SCORE]))
        if cnv_iid in cnv.keys():
            cnv[cnv_iid].append(add_cnv)
        else:
            cnv[cnv_iid] = [add_cnv]
    else:
        blank_dgr += 1

print(len(cnv), (blank_dgr))
print([k for k in cnv.keys()[:5]])

names_from = ["CHR de Merge_CIA_610_660_QC", 'START', 'STOP']
cnv_names = cnv_uiid(arr, names_from)
print("with names from: ", names_from)
print("we have {} unique elements out of {} rows in the tsv".format(
                                len(np.unique(cnv_names)), len(cnv_names)))
#print(cnv_names[:5])

# To be check : the 7337 unique cnv should reduce to 3374 when we remove those for which dgr == ''

__kcnv = cnv.keys()
print(len(__kcnv))
for _ in __kcnv[3330:3350]:
    print(_,': ',cnv[_])

#for k in cnv.keys():
#    if len(cnv[k]) > 1:
#        if len(set([cnv[k][i][0] for i in range(len(cnv[k]))])) > 1:
#            pass
#            # print(cnv[k])
#        #print([t[0] for t in cnv[k]])


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
Create a dictionary of the subjects -
Can you write Python code for it?

#print(len(arr), line0)

cnv = OrderedDict({})
#names_from = ['START', 'STOP', "5'gene", "3'gene", "5'dist(kb)", "3'dist(kb)"]
names_from = ['IID_projet']

for line in arr[1:]:
    lline = line.split('\t')
    dgr = lline[i_DANGER]
    scr  = lline[i_SCORE]
    sub_iid = cnv_uiid(line, names_from, arr[0])

    if dgr != '':
        add_cnv = (danger_score(lline[i_DANGER], pH1_with_apriori),
                   score2prob(lline[i_SCORE]))
        if sub_iid in cnv.keys():
            cnv[sub_iid].append(add_cnv)
        else:
            cnv[sub_iid] = [add_cnv]


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
Finally, putting things together
Can you write Python code for it?

# re-loading
CWD = osp.join(osp.expanduser('~'), 'documents','grants_projects','roberto_projects', \
               'guillaume_huguet_CNV','File_OK')
filename = 'Imagen_QC_CIA_MMAP_V2_Annotation.tsv'
fullfname = osp.join(CWD, filename)

# in numpy array
arr = np.loadtxt(fullfname, dtype='str', comments=None, delimiter='\Tab',
                 converters=None, skiprows=0, usecols=None, unpack=False, ndmin=0)

line0 = arr[0].split('\t')
DANGER = 'Pvalue_MMAP_V2_sans_intron_and_Intergenic'
SCORE = 'SCORE'

i_DANGER = line0.index(DANGER)
i_SCORE = line0.index(SCORE)
i_START = line0.index('START')
i_STOP = line0.index('STOP')
i_5pGENE = line0.index("5'gene")
i_3pGENE = line0.index("3'gene")
i_5pDIST = line0.index("5'dist(kb)")
i_3pDIST = line0.index("3'dist(kb)")
#i_LOC = line0.index('Location')

for idx in range(len(line0)):
    print(line0[idx],': ', arr[1].split('\t')[idx])

# A function to create unique identifiers of subj, CNV, etc

def cnv_uiid(tsv_arr, columns_names, first_line=None):
    """
    columns_names: list of column names with which the uiid are constructed
    returns a list if given the full array,
    returns a string if given only one line (and the first line)
    """

    chr2rm = ''.join([',', '.', ' ']) # others: ['!', '?', ...] ?

    if isinstance(tsv_arr, six.string_types) and (first_line != None):
        # assume we are given one line of the tsv array
        indexes = [first_line.split('\t').index(colname) for colname in columns_names]
        uiid = '_'.join([(tsv_arr.split('\t')[ind]).translate(None,chr2rm) for ind in indexes])

    else:
        uiid = []
        indexes = [tsv_arr[0].split('\t').index(colname) for colname in columns_names]
        for line in tsv_arr[1:]:
            ll = line.split('\t')
            uiid.append('_'.join([ll[ind] for ind in indexes]))

    return uiid

print(line0)

#names_from = ['START', 'STOP', "5'gene", "3'gene", "5'dist(kb)", "3'dist(kb)"]

#---------- ligne uniques:
names_from = ['IID_projet', 'IID_genotype', "CHR de...


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
Histogram of the number of cnv used to compute dangerosity
Can you write Python code for it?


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
Printing out results
Can you write Python code for it?

outfile = 'dangerosity_cnv.txt'
fulloutfile = osp.join(CWD, outfile)

with open(fulloutfile, 'w') as outf:
    for sub in cnv:
        outf.write("\t".join([sub, str(dangerosity(cnv[sub]))]) + "\n")


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
QUESTION Guillaume
The last interval -- the one for which proba is 95.0 -- is [30.0, 32.5) ?
Creating a function from score to proba from Guillaume's values
A piece wise linear function
Can you write Python code for it?

# Solution 1 : affine by pieces
def aff(bi, bs, vbi, vbs, x0, y0):
    """
    returns the affine function
    """
    a = (vbs - vbi)/(bs - bi)
    b = y0  - a*x0

    def this_aff(x):
        return a*x + b

    return this_aff

def create_score2prob(p_cnv):

    # create a dict that has the affine function
    pf_cnv = OrderedDict({})
    kcnv = p_cnv.keys()

    for idx,k in enumerate(kcnv[:-1]):
        k1 = kcnv[idx+1]
        pf_cnv[k] = aff(k, k1, p_cnv[k], p_cnv[k1], k, p_cnv[k])

    # testing just this
    def test_pf_cnv():
        assert pf_cnv[15](15) == .20
        assert abs(pf_cnv[15](17.5) - pf_cnv[17.5](17.5)) < np.finfo(float).eps
        assert abs(pf_cnv[17.5](20) - pf_cnv[20.](20.)) < np.finfo(float).eps
        assert abs(pf_cnv[20.](22.5) - pf_cnv[22.5](22.5)) < np.finfo(float).eps
        #assert pf_cnv[30](45) == 1.
        return True
    test_pf_cnv()

    # function linear piecewise:
    def sc2prob(sc):
        """
        returns the proba that cnv with score sc is a cnv;
        """
        #
        assert sc >= 15.
        kcnv = pf_cnv.keys()

        # if greater than the last key, returns 1.0
        if sc >= kcnv[-1]:
            return 1.0

        idx = 0
        while sc > kcnv[idx+1]: idx += 1

        return pf_cnv[kcnv[idx]](sc)

    return sc2prob


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
QUESTION Guillaume :
25 -> 27.5 : 83% true
27.5 -> 30 : 77% true
Y a t-il eu inversion ? ou s'agit il de bruit ? Reponse : il s'agit de bruit.
Can you write Python code for it?

from collections import OrderedDict

p_cnv = OrderedDict({})
for line in prob_cnv_vrai.splitlines():
    line = line.split('\t')
    # print(line)
    p_cnv[float(line[0].replace(',','.'))] = float(line[1])/100.

# if model with linear piecewise
# p_cnv[45.] = 1.
# CORRECTION for a possible inversion - or simply "smooth" the curve !
# p_cnv[25], p_cnv[27.5] = p_cnv[27.5] , p_cnv[25]

print(p_cnv)