I am working on the file "DAT236x Deep Learning Explained/Lab6_TextClassification_with_LSTM.ipynb".
The first task is:
Task and Model Structure
The task we want to approach in this tutorial is slot tagging.
We use the ATIS corpus.
ATIS contains human-computer queries from the domain of Air Travel Information Services,
and our task will be to annotate (tag) each word of a query with the specific item of information (slot) it belongs to, if any.
The data in your working folder has already been converted into a CTF (CNTK Text Format) file.
Let us look at an example from the test-set file atis.test.ctf:
19  |S0 178:1 |# BOS      |S1 14:1 |# flight  |S2 128:1 |# O
19  |S0 770:1 |# show                         |S2 128:1 |# O
19  |S0 429:1 |# flights                      |S2 128:1 |# O
19  |S0 444:1 |# from                         |S2 128:1 |# O
19  |S0 272:1 |# burbank                      |S2 48:1  |# B-fromloc.city_name
19  |S0 851:1 |# to                           |S2 128:1 |# O
19  |S0 789:1 |# st.                          |S2 78:1  |# B-toloc.city_name
19  |S0 564:1 |# louis                        |S2 125:1 |# I-toloc.city_name
19  |S0 654:1 |# on                           |S2 128:1 |# O
19  |S0 601:1 |# monday                       |S2 26:1  |# B-depart_date.day_name
19  |S0 179:1 |# EOS                          |S2 128:1 |# O

This file has 5-7 columns per line (each separated by the "|" character):

a sequence id (19). There are 11 entries with this sequence id. This means that sequence 19 consists
of 11 tokens;
column S0, which contains numeric word indices; the input data is encoded in one-hot vectors.  There are 943 words in the vocabulary, so each word is a 943 element vector of all 0 with a 1 at a vector index chosen to represent that word.  For example the word "from" is represented with a 1 at index 444 and zero everywhere else in the vector. The word "monday" is represented with a 1 at index 601 and zero everywhere else in the vector.
a comment column denoted by #, to allow a human reader to know what the numeric word index stands for;
Comment columns are ignored by the system. BOS and EOS are special words
to...
Can you write Python code for it?

# setting seed
np.random.seed(0)
C.cntk_py.set_fixed_random_seed(1)
C.cntk_py.force_deterministic_algorithms()

# number of words in vocab, slot labels, and intent labels
vocab_size = 943 ; num_labels = 129 ; num_intents = 26

# model dimensions
input_dim  = vocab_size
label_dim  = num_labels

emb_dim    = 150 # error 2.25%
# emb_dim    = 50 # error  2.71%
#emb_dim = 300  # error 2.20%

#hidden_dim = 100 # error 2.44%
hidden_dim = 300 # error 2.25%
#hidden_dim = 500 # error 2.09%

# Create the containers for input feature (x) and the label (y)
x = C.sequence.input_variable(vocab_size)
y = C.sequence.input_variable(num_labels)

def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim, name='embed'),
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_labels, name='classify')
        ])


I am working on the file "DAT236x Deep Learning Explained/Lab6_TextClassification_with_LSTM.ipynb".
The first task is:
This shows how learning proceeds over epochs (passes through the data).
For example, after four epochs, the loss, which is the cross-entropy criterion, has reached 0.11 as measured on the ~18000 samples of this epoch,
and that the error rate is 2.6% on those same 18000 training samples.
The epoch size is the number of samples--counted as word tokens, not sentences--to
process between model checkpoints.
Once the training has completed (a little less than 2 minutes on a Titan-X or a Surface Book),
you will see an output like this
Finished Epoch [10]: [Training] loss = 0.033263 * 18039, metric = 0.9% * 18039
which is the loss (cross entropy) and the metric (classification error) averaged over the final epoch.
On a CPU-only machine, it can be 4 or more times slower. You can try setting
python
emb_dim    = 50
hidden_dim = 100
to reduce the time it takes to run on a CPU, but the model will not fit as well as when the
hidden and embedding dimension are larger.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab6_TextClassification_with_LSTM.ipynb".
The first task is:
Training the model
We are using the Progress Printer to display the training loss and classification error throughout training epochs.
The training should take less than 2 minutes on a Titan-X or a Surface Book. Once the training completed, you will see an output like this
Finished Epoch [10]: [Training] loss = 0.033263 * 18039, metric = 0.9% * 18039
which is the loss (cross entropy) and the metric (classification error) averaged over the final epoch.
On a CPU-only machine, it can be 4 or more times slower. You can try setting
python
emb_dim    = 50
hidden_dim = 100
to reduce the time it takes to run on a CPU, but the model will not fit as well as when the
hidden and embedding dimension are larger.
Testing the model
We also use the Progress Printer to display the accuracy on a test set by computing the error over multiple minibatches of test data. For evaluating on a small sample read from a file, you can set a minibatch size reflecting the sample size and run the test_minibatch on that instance of data. To see how to evaluate a single sequence, we provide an instance later in the tutorial.
Can you write Python code for it?

def train_test(train_reader, test_reader, model_func, max_epochs=10):

    # Instantiate the model function; x is the input (feature) variable
    model = model_func(x)

    # Instantiate the loss and error function
    loss, label_error = create_criterion_function_preferred(model, y)

    # training config
    epoch_size = 18000        # 18000 samples is half the dataset size
    minibatch_size = 70

    # LR schedule over epochs
    # In CNTK, an epoch is how often we get out of the minibatch loop to
    # do other stuff (e.g. checkpointing, adjust learning rate, etc.)
    # (we don't run this many epochs, but if we did, these are good values)
    lr_per_sample = [0.003]*4+[0.0015]*24+[0.0003]
    lr_per_minibatch = [lr * minibatch_size for lr in lr_per_sample]
    lr_schedule = C.learning_rate_schedule(lr_per_minibatch, C.UnitType.minibatch, epoch_size)

    # Momentum schedule
    momentum_as_time_constant = C.momentum_as_time_constant_schedule(700)

    # adam error 2.25%
    # sgd error 10%
    # fsadagrad 3.16%

    # We use a the Adam optimizer which is known to work well on this dataset
    # Feel free to try other optimizers from
    # https://www.cntk.ai/pythondocs/cntk.learner.html#module-cntk.learner
    learner = C.adam(parameters=model.parameters,
                     lr=lr_schedule,
                     momentum=momentum_as_time_constant,
                     gradient_clipping_threshold_per_sample=15,
                     gradient_clipping_with_truncation=True)

#     learning_rate = 0.2
#     lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)
#     learner = C.sgd(z.parameters, lr_schedule)

#     learning_rate = 0.2
#     lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)
#     TIMESTEPS = 14
#     BATCH_SIZE = TIMESTEPS * 10
#     momentum_time_constant = C.momentum_as_time_constant_schedule(BATCH_SIZE / -math.log(0.9))
#     learner = C.fsadagrad(z.parameters,
#                       lr = lr_schedule,
#      ...


I am working on the file "DATA/ELC_computable_report_AY1920.ipynb".
The first task is:
Calendar heat map of sign-ins
The ELC was less utilized by students in the Spring semester
Can you write Python code for it?

#@title
import calmap #The Calendar (heat) map package is imported to provide higher quality visualization than bargraphs
#https://pythonhosted.org/calmap/#'
#plt.figure(figsize=(12,18))
#calmap.yearplot(df_mdy, year=2019, daylabels='MTWTRFSS')
#plt.show()

print('Calendar Map of ELC Sign-ins for Fall 2019')
fig = plt.figure(figsize=(12,5))
ax = fig.add_subplot(111)
cax = calmap.yearplot(obj_mdy, year=2019, ax=ax, cmap='jet')
fig.colorbar(cax.get_children()[1], ax=cax, orientation='horizontal')

print('\n\n\n')

print('Calendar Map of ELC Sign-ins for Spring 2020')
fig = plt.figure(figsize=(12,5))
ax = fig.add_subplot(111)
cax = calmap.yearplot(obj_mdy, year=20, ax=ax, cmap='jet')
fig.colorbar(cax.get_children()[1], ax=cax, orientation='horizontal')


I am working on the file "DATA/ELC_computable_report_AY1920.ipynb".
The first task is:
Grouping of data by (visits, day, hour): 3D scatter plot
Financial data
Cost per hour and number of coaches per hour, in any given regular week (week 2 - week before final exams).  A zero or NaN entry in a table suggests that the ELC is not open during those hours.
The actual ELC expenditure on coach salaries was recorded as $26,041.38 for AY 2019-2020.  The initial estimate was $27618.13.  Since it took 1 week to restart the ELC after COVID19 closure, this reflected as approximately ~$1577.  The ELC coach salaries amount to approximately ~$1000 a week.
Can you write Python code for it?

#@title
#url4 = 'https://raw.githubusercontent.com/dnaneet/ELC/master/DATA/expenses_f19.csv'
#df4 = pd.read_csv(url4, error_bad_lines=False)
df4 = pd.read_csv('expenses_ay1920.csv', error_bad_lines=False)
print('ELC Expenses ($) on wages for AY 19-20')
print(tabulate(df4, headers='keys', tablefmt='psql'))

print('\n\n\n')

url2='https://raw.githubusercontent.com/dnaneet/ELC/master/DATA/ay1920_costperhour.csv'
df2 = pd.read_csv(url2)
print('Average cost ($) per hour by hour of day, for the semester')
print(tabulate(df2, headers='keys', tablefmt='psql'))
#print('Mean cost per day ($):\n')
#print(df2.mean())

print('\n\n\n')

print('Average (rounded) number of coaches per hour')
url3='https://raw.githubusercontent.com/dnaneet/ELC/master/DATA/ay1920_num_coaches_per_hr.csv'
df3 = pd.read_csv(url3)
print(tabulate(df3, headers='keys', tablefmt='psql'))


I am working on the file "DATA/ELC_computable_report_AY1920.ipynb".
The first task is:
Sign-ins by course:
Statics (MEEM2110) help was the most used service.  Since 2018, it has been an ELC policy to ensure that all coaches hired are proficient in Statics.
Can you write Python code for it?

#@title
obj_course = df['visits'].groupby(df['visits']).count()

course_data = np.array(obj_course)
#df_course = pd.DataFrame({'course': ['Statics', 'MechanicsOfMatls', 'Thermodynamics','Dynamics', 'MATLAB', 'StudySpace'],
#                          'NumberOfSignins' : course_data})

#print(course_data)
df_course = pd.DataFrame({'course': ['Statics', 'MechanicsOfMatls', 'Thermodynamics','Dynamics', 'MATLAB', 'StudySpace'],
                          'NumberOfSignIns': course_data[0:6]})
df_course.plot.bar(x='course', y='NumberOfSignIns', rot=90,figsize=(12,8))
plt.show()

fig = px.bar(df_course, x="course", y="NumberOfSignIns")
fig.show()


I am working on the file "DATA/ELC_computable_report_AY1920.ipynb".
The first task is:
Sign-ins by hour of day
Afternoon hours were the most popular time (pre-COVID19) for students to visit the ELC.  Since 2018, ELC has been staffed with ~50% more coaches during the 12-3pm time period, as compared to other periods.
Can you write Python code for it?

#@title
obj_hr = df['visits'].groupby(df['hour']).count() #Grouping number of visits by hour
xdata = np.array(obj_hr.reset_index())[:,0]
ydata = np.array(obj_hr.reset_index())[:,1]

df_hr = pd.DataFrame({'hour': xdata, 'visits': ydata})
fig = px.bar(df_hr, x="hour", y="visits")
fig.show()

df_hr.plot.bar(x='hour', y='visits', rot=90,figsize=(12,8))
plt.show()
#df_hr.head(10) #What does this grouped data look like?
#plt.xlabel('Hour of day')
#plt.ylabel('Total number of sign-ins')

#ax2 = df_hr.plot(kind='bar')
#plt.plot()


I am working on the file "DATA/ELC_computable_report_AY1920.ipynb".
The first task is:
Sign-ins by month
Since 2018, the month of September has had the most sign-ins.  This is due to a week-2 influx of Mechanics of Materials student visits for Statics help.
Can you write Python code for it?

#@title
obj_month = df['visits'].groupby(df['month']).count() #Grouping number of visits by day
#print(df_month.head(10)) #What does this grouped data look like?

xdata = np.array(obj_month.reset_index())[:,0]
ydata = np.array(obj_month.reset_index())[:,1]

df_month = pd.DataFrame({'month': xdata, 'visits': ydata})
df_month["monthName"]=['Jan', 'Feb', 'Mar','Sep', 'Oct', 'Nov', 'Dec']

fig = px.bar(df_month, x="monthName", y="visits")
fig.show()

df_month.plot.bar(x='monthName', y='visits', rot=90,figsize=(12,8))
plt.show()

#ax = dobj_month.plot(kind='bar')
#ax.set_xticklabels(df_mdy.tail(n).index.strftime('%b-%d-%y'));
#ax.set_xticks(np.arange(df_month.shape[0]), ('Jan', 'Feb', 'Mar', 'Apr', 'Jun', 'Aug', 'Sep', 'Nov'));
#plt.xticks(np.arange(obj_month.shape[0]), ('Sep', 'Oct', 'Nov', 'Dec'), rotation=90)
#plt.xlabel('Month')
#plt.ylabel('Number of sign-ins')
#plt.show()


I am working on the file "DATA/ELC_computable_report_AY1920.ipynb".
The first task is:
Visualization of ELC usage data
Now that the ELC visit data has been cast into the appropriate format, exploratory visualization is performed. The exploratory steps are alongside lines of code, as comments.
Sign-ins by date
The ELC has peak usage in the 2nd or 3rd week of September.  This can be correlated to Mechanics of Materials students requiring help with Statics
Can you write Python code for it?

#@title
n=5 #last n records
obj_mdy = df['visits'].groupby(df['mdy']).count() #Grouping number of visits by day
#obj_mdy.tail(n) #What does this grouped data look like?

xdata = np.array(obj_mdy.reset_index())[:,0]
ydata = np.array(obj_mdy.reset_index())[:,1]

df_mdy = pd.DataFrame({'mdy': xdata, 'visits': ydata})

fig = px.bar(df_mdy, x="mdy", y="visits")
fig.show()
#df_mdy.sample(n=5, random_state=1)
#df_day_hour['dayName'] = df_day_hour['day'].replace({1:'Mon', 2:'Tue', 3:'Wed', 4:'Thur', 5:'Fri', 6:'Sat', 7:'Sun'}) #replace day names with day numbers
#ax = obj_mdy.tail(len(df)).plot(kind='bar')
#ax.set_xticklabels(obj_mdy.tail(len(df)).index.strftime('%b-%d-%y'));
#plt.xlabel('Date')
#plt.ylabel('Number of sign-ins')
#plt.show()

#df_mdy.head(5)
df_mdy['DayOfOperation'] = np.arange(len(df_mdy))

df_mdy.plot.bar(x='DayOfOperation', y='visits', rot=90,figsize=(12,8))
plt.show()


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Analyse: CITY

Wie sollte die Spalte verwendet werden?
In der Spalte kommen auch Städte mit Leerzeichen, Minus, Klammer und Slash vor
Zunächst wird die Information als Attribut umgesetzt.
Can you write Python code for it?

import functools as ft
ft.reduce( \
    lambda x, y: x & y , \
        map( \
            lambda x: len(x.split())==1 \
                , df_stations['CITY'].unique()) \
    )

list(filter( \
    lambda x: len(x.split())>1 \
    , df_stations['CITY'].unique() \
))


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Analyse: RENTAL_ZONE_HAL_SRC

Hat die Spalte eine Relevanz?
Da in der Spalte nur ein Wert vorkommt: Nicht
Can you write Python code for it?


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Analyse: TYPE

Hat die Spalte eine Relevanz?
Ja, die Spalte eignet sich als Label
Can you write Python code for it?

df_stations['TYPE'].unique()

df_stations['LABEL'] = df_stations.apply(getLabel, axis=1, nodeName='RENTAL_ZONE')
df_stations.head()


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Aufnehmen der Buchungsinformationen
Can you write Python code for it?

#List unique values in the df['name'] column
df_booking.TECHNICAL_INCOME_CHANNEL.unique()

df_booking['INCOME_CHANNEL_NAME'] = df_booking.agg({'TECHNICAL_INCOME_CHANNEL' : lambda x: trimName(x)})

df_booking.INCOME_CHANNEL_NAME.head()

df_booking.set_index("BOOKING_HAL_ID", inplace= True)
df_booking.head()


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Die Tarif-Informationen aufnehmen

Vorgehensweise:
- Wir wollen auch ein Label für die Buchungen realisieren, um für die Analyse eine einfacheres Mittel zu haben.
 - Aus dem Grund macht es Sinn, neben den Tarifen auch Haupttarife zu definieren, die dann in einer Buchung als Label aufgeführt wird.
  - Deswegen sind folgende Schritte notwendig
   - Erster Schritt ist die Reduzierung der Tarifbezeichung auf ein Wort (als Name des Haupttarifs)
   - Zweiter Schritt ist die Aufnahme in Buchung als zusätzliche Spalte
- Tarif und Haupttarife haben ein festes Label
- Teilen sich denselben Nummernsraum als Primärschlüssel

Bearbeitung der Tarifinformationen

Label-Info erzeugen
Bezeichnung für die Haupttarif erzeugen
Label für das Haupttarif erzeugen
IDs für die Haupttarife erstellen
Typ der Beziehung zwische Tarif und Haupttarif erstellen
Can you write Python code for it?

df_cat['LABEL'] = df_cat.apply(getLabel, axis=1, nodeName='CATEGORY')
df_cat.head()

df_cat['PARENT_CATEGORY'] = df_cat.agg({'CATEGORY' : lambda x: x.split()[0].replace('/', '')})
df_cat.head()

df_cat['PARENT_LABEL'] = df_cat.apply(getLabel, axis=1, nodeName='PARENT_CATEGORY')
df_cat.head()

df_cat['PARENT_ID'] = df_cat.agg({'HAL_ID' : lambda x: x + 10000})
df_cat.head()

df_cat['REL_TYPE_CAT_PCAT'] = df_cat.apply(getRelationshipType, axis=1, types=["CATEGORY","PARENT_CATEGORY"])
df_cat.head()


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Header für die Station-Elemente

rentalZoneID:ID(RENTAL-ZONE-ID)|name|code|type|city|latitude:float|longtitude:float|poiAirpor|poiLongDistanceTrains|poiSuburbanTrains|poiUnderground|:LABEL

Ausgeben der Quelldateien für Neo4J-Import

Stationen
Can you write Python code for it?

# Die Nodes für die Fahrzeuge
writeDsvFile(df_stations, 'rentalZones', '|', \
             ['RENTAL_ZONE_HAL_ID', 'NAME', 'CODE', 'TYPE', 'CITY', 'LATITUDE', \
                  'LONGTITUDE', 'POI_AIRPORT_X', 'POI_LONG_DISTANCE_TRAINS_X', 'POI_SUBURBAN_TRAINS_X', \
                  'POI_UNDERGROUND_X', 'LABEL']\
             , [ 'rentalZoneID:ID(RENTAL-ZONE-ID)', 'name', 'code', 'type', 'city', 'latitude:float', \
                    'longtitude:float', 'poiAirport', 'poiLongDistanceTrains', 'poiSuburbanTrains', \
                    'poiUnderground', ':LABEL'])


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Lesen der Buchungsinformationen
Can you write Python code for it?

#df_cat.set_index("HAL_ID", inplace= True)
#df_cat.head()

df_booking = pd.read_csv('./datasets/OPENDATA_BOOKING_CARSHARING.csv', quotechar='"',encoding ='utf-8', sep=';')
df_booking.head()


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Lesen der Fahrzeuginformationen
Can you write Python code for it?

df_fz = pd.read_csv('./datasets/OPENDATA_VEHICLE_CARSHARING.csv', quotechar='"',encoding ='utf-8', sep=';')
df_fz.head()


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Lesen der Tarif-Informationen
Can you write Python code for it?

df_cat = pd.read_csv('./datasets/OPENDATA_CATEGORY_CARSHARING.csv', quotechar='"',encoding ='utf-8', sep=';')
df_cat.head()


I am working on the file "DB_OpenData/.ipynb_checkpoints/Carsharing_Data-checkpoint.ipynb".
The first task is:
Technical income channel
Möglicher Umgang mit den Informationen über die Quelle (Technical income channel) der Bestellung:

Als Node mit weiteren Informationen/Beziehungen:
- Es liegen Detail-Informationen über die Quelle vor und die Informationen sind fachlich in Bezug auf die Haltung der Datenrelevant. Notwendige Aufwände:
  - Integrieren der Detail-Informationen
  - ID-Namespace für die Nodes
  - Klärung der notwendige Beziehungen
Als Label in einer Buchung:
- Es liegen keine Detail-Informationen über die Quelle vor. Notwendige Aufwände:
  - Die Quellbenennungen Label-Fähig machen (Reduktion auf ein Wort)
  - Die Labelnamen als zusätzliche Spalte in df_booking aufnehmen.

Referenzierte Informationen in einer Buchung:

CATEGORY_HAL_ID -> Kategorie
VEHICLE_HAL_ID -> Fahrzeug
CUSTOMER_HAL_ID -> Kunde (Datensätze nicht verfügbar)
START_RENTAL_ZONE_HAL_ID -> Abholstation
END_RENTAL_ZONE_HAL_ID -> Rückgabestation
Can you write Python code for it?

df_cat_labels = df_cat.drop(['CATEGORY', 'COMPANY', 'COMPANY_GROUP', 'PARENT_LABEL', 'LABEL', 'PARENT_ID', 'REL_TYPE_CAT_PCAT'], axis=1)
df_booking_cat = pd.merge(df_booking, df_cat_labels, left_on='CATEGORY_HAL_ID', right_on='HAL_ID') \
                    .drop(['CUSTOMER_HAL_ID','HAL_ID','RENTAL_ZONE_HAL_SRC'], axis=1)
df_booking_cat.head()


I am working on the file "DC Council Calendar Web Scrape.ipynb".
The first task is:
Below is the code you would run if you do not run the previous 3 blocks of code (When in production mode and no longer needing historical data).
Can you write Python code for it?

# url = 'http://dccouncil.us/calendar'
# r = requests.get(url)
# soup = BeautifulSoup(r.text, 'lxml')


I am working on the file "DC Council Calendar Web Scrape.ipynb".
The first task is:
Create DataFrame for data and export to csv
Can you write Python code for it?

data = {'Date': date,
       'Time' : time,
       'Location': location,
       'Titles': titles,
       'Content': content}

df = pd.DataFrame(data, columns = ['Date', 'Time', 'Location', 'Titles', 'Content'])


I am working on the file "DC Council Calendar Web Scrape.ipynb".
The first task is:
Extract from div class: event-description-content-dev for title and contents of meeting
Can you write Python code for it?

titles = [div.find('a').text for div in soup.find_all('div', {'class': 'event-description-content-dev'})]

titles

children = soup.findChildren('div', {'class': 'event-description-content-dev'})

content = []
for item in children:
    # Don't need Title info as that's already been extracted. Start reading content at index = 1 and extract data to end
    item = [content for content in item.text.split('\n') if len(content)>0][1:]

    # Create string from separate list items to all be listed in
    item = ' '.join(item)
    content.append(item)

content


I am working on the file "DC Council Calendar Web Scrape.ipynb".
The first task is:
Review underlying html structure of webpage
Can you write Python code for it?


I am working on the file "DC Council Calendar Web Scrape.ipynb".
The first task is:
Set Date
By default, this page loads with the current date. The calendar does not always show any data for the current date/week. For the initial code, I have set a date to Jun 1 to grab historical data up to the current date.
Note: when deploying as script to run daily, code here can be deleted and different soup object created
Can you write Python code for it?

date = '2017-06-01'
browser.find_element_by_id('cal-date-select-dev').clear()
browser.find_element_by_id('cal-date-select-dev').send_keys(date)
browser.find_element_by_xpath('//td//input[contains(@class, "site-button-small-dev submit")]').click()


I am working on the file "DC Council Calendar Web Scrape.ipynb".
The first task is:
Specify URL from which to scrape, create browser object
Can you write Python code for it?

url = ('http://dccouncil.us/calendar')
browser = webdriver.Firefox()
browser.get(url)

wait = WebDriverWait(browser, 30)


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
And another function we can use to train our network. Notice when we call generator to create the samples to display, we set training to False. That's so the batch normalization layers will use the population statistics rather than the batch statistics. Also notice that we set the net.input_real placeholder when we run the generator's optimizer. The generator doesn't actually use it, but we'd get an errror without it because of the tf.control_dependencies block we created in model_opt.
Can you write Python code for it?

def train(net, dataset, epochs, batch_size, print_every=10, show_every=100, figsize=(5,5)):
    saver = tf.train.Saver()
    sample_z = np.random.uniform(-1, 1, size=(72, z_size))

    samples, losses = [], []
    steps = 0

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        for e in range(epochs):
            for x, y in dataset.batches(batch_size):
                steps += 1

                # Sample random noise for G
                batch_z = np.random.uniform(-1, 1, size=(batch_size, z_size))

                # Run optimizers
                _ = sess.run(net.d_opt, feed_dict={net.input_real: x, net.input_z: batch_z})
                _ = sess.run(net.g_opt, feed_dict={net.input_z: batch_z, net.input_real: x})

                if steps % print_every == 0:
                    # At the end of each epoch, get the losses and print them out
                    train_loss_d = net.d_loss.eval({net.input_z: batch_z, net.input_real: x})
                    train_loss_g = net.g_loss.eval({net.input_z: batch_z})

                    print("Epoch {}/{}...".format(e+1, epochs),
                          "Discriminator Loss: {:.4f}...".format(train_loss_d),
                          "Generator Loss: {:.4f}".format(train_loss_g))
                    # Save losses to view after training
                    losses.append((train_loss_d, train_loss_g))

                if steps % show_every == 0:
                    gen_samples = sess.run(
                                   generator(net.input_z, 3, reuse=True, training=False),
                                   feed_dict={net.input_z: sample_z})
                    samples.append(gen_samples)
                    _ = view_samples(-1, samples, 6, 12, figsize=figsize)
                    plt.show()

        saver.save(sess, './checkpoints/generator.ckpt')

    with open('samples.pkl', 'wb') as f:
        pkl.dump(samples, f)

    return losses, samples


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
Building the model
Here we can use the functions we defined about to build the model as a class. This will make it easier to move the network around in our code since the nodes and operations in the graph are packaged in one object.
Can you write Python code for it?

class GAN:
    def __init__(self, real_size, z_size, learning_rate, alpha=0.2, beta1=0.5):
        tf.reset_default_graph()

        self.input_real, self.input_z = model_inputs(real_size, z_size)

        self.d_loss, self.g_loss = model_loss(self.input_real, self.input_z,
                                              real_size[2], alpha=0.2)

        self.d_opt, self.g_opt = model_opt(self.d_loss, self.g_loss, learning_rate, 0.5)


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
Generator
Here you'll build the generator network. The input will be our noise vector z as before. Also as before, the output will be a $tanh$ output, but this time with size 32x32 which is the size of our SVHN images.
What's new here is we'll use convolutional layers to create our new images. The first layer is a fully connected layer which is reshaped into a deep and narrow layer, something like 4x4x1024 as in the original DCGAN paper. Then we use batch normalization and a leaky ReLU activation. Next is a transposed convolution where typically you'd halve the depth and double the width and height of the previous layer. Again, we use batch normalization and leaky ReLU. For each of these layers, the general scheme is convolution > batch norm > leaky ReLU.
You keep stack layers up like this until you get the final transposed convolution layer with shape 32x32x3. Below is the archicture used in the original DCGAN paper:

Note that the final layer here is 64x64x3, while for our SVHN dataset, we only want it to be 32x32x3.
Can you write Python code for it?

def generator(z, output_dim, reuse=False, alpha=0.2, training=True):
    with tf.variable_scope('generator', reuse=reuse):
        # First fully connected layer
        x1 = tf.layers.dense(z, 4*4*512)
        # Reshape it to start the convolutional stack
        x1 = tf.reshape(x1, (-1, 4, 4, 512))
        x1 = tf.layers.batch_normalization(x1, training=training)
        x1 = tf.maximum(alpha * x1, x1)
        # 4x4x512 now

        x2 = tf.layers.conv2d_transpose(x1, 256, 5, strides=2, padding='same')
        x2 = tf.layers.batch_normalization(x2, training=training)
        x2 = tf.maximum(alpha * x2, x2)
        # 8x8x256 now

        x3 = tf.layers.conv2d_transpose(x2, 128, 5, strides=2, padding='same')
        x3 = tf.layers.batch_normalization(x3, training=training)
        x3 = tf.maximum(alpha * x3, x3)
        # 16x16x128 now

        # Output layer
        logits = tf.layers.conv2d_transpose(x3, output_dim, 5, strides=2, padding='same')
        # 32x32x3 now

        out = tf.tanh(logits)

        return out


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
Hyperparameters
GANs are very senstive to hyperparameters. A lot of experimentation goes into finding the best hyperparameters such that the generator and discriminator don't overpower each other. Try out your own hyperparameters or read the DCGAN paper to see what worked for them.
Can you write Python code for it?

real_size = (32,32,3)
z_size = 100
learning_rate = 0.0002
batch_size = 128
epochs = 25
alpha = 0.2
beta1 = 0.5

# Create the network
net = GAN(real_size, z_size, learning_rate, alpha=alpha, beta1=beta1)

dataset = Dataset(trainset, testset)

losses, samples = train(net, dataset, epochs, batch_size, figsize=(10,5))

fig, ax = plt.subplots()
losses = np.array(losses)
plt.plot(losses.T[0], label='Discriminator', alpha=0.5)
plt.plot(losses.T[1], label='Generator', alpha=0.5)
plt.title("Training Losses")
plt.legend()

fig, ax = plt.subplots()
losses = np.array(losses)
plt.plot(losses.T[0], label='Discriminator', alpha=0.5)
plt.plot(losses.T[1], label='Generator', alpha=0.5)
plt.title("Training Losses")
plt.legend()

_ = view_samples(-1, samples, 6, 12, figsize=(10,5))

_ = view_samples(-1, samples, 6, 12, figsize=(10,5))


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
MNIST database
The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems. The database is also widely used for training and testing in the field of machine learning.
The MNIST database contains 60,000 training images and 10,000 testing images. Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset.
Can you write Python code for it?

(X_train, y_train), (X_test, y_test) = mnist.load_data()

x_train = input_data.read_data_sets("mnist",one_hot=True).train.images
x_train = x_train.reshape(-1, 28,28, 1).astype(np.float32)

print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

pixels = x_train[0]
pixels = pixels.reshape((28, 28))
# Plot
plt.imshow(pixels, cmap='gray')

plt.show()


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
Model Loss
Calculating the loss like before, nothing new here.
Can you write Python code for it?

def model_loss(input_real, input_z, output_dim, alpha=0.2):
    """
    Get the loss for the discriminator and generator
    :param input_real: Images from the real dataset
    :param input_z: Z input
    :param out_channel_dim: The number of channels in the output image
    :return: A tuple of (discriminator loss, generator loss)
    """
    g_model = generator(input_z, output_dim, alpha=alpha)
    d_model_real, d_logits_real = discriminator(input_real, alpha=alpha)
    d_model_fake, d_logits_fake = discriminator(g_model, reuse=True, alpha=alpha)

    d_loss_real = tf.reduce_mean(
        tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_real, labels=tf.ones_like(d_model_real)))
    d_loss_fake = tf.reduce_mean(
        tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_fake, labels=tf.zeros_like(d_model_fake)))
    g_loss = tf.reduce_mean(
        tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_fake, labels=tf.ones_like(d_model_fake)))

    d_loss = d_loss_real + d_loss_fake

    return d_loss, g_loss


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
Optimizers
Not much new here, but notice how the train operations are wrapped in a with tf.control_dependencies block so the batch normalization layers can update their population statistics.
Can you write Python code for it?

def model_opt(d_loss, g_loss, learning_rate, beta1):
    """
    Get optimization operations
    :param d_loss: Discriminator loss Tensor
    :param g_loss: Generator loss Tensor
    :param learning_rate: Learning Rate Placeholder
    :param beta1: The exponential decay rate for the 1st moment in the optimizer
    :return: A tuple of (discriminator training operation, generator training operation)
    """
    # Get weights and bias to update
    t_vars = tf.trainable_variables()
    d_vars = [var for var in t_vars if var.name.startswith('discriminator')]
    g_vars = [var for var in t_vars if var.name.startswith('generator')]

    # Optimize
    with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):
        d_train_opt = tf.train.AdamOptimizer(learning_rate, beta1=beta1).minimize(d_loss, var_list=d_vars)
        g_train_opt = tf.train.AdamOptimizer(learning_rate, beta1=beta1).minimize(g_loss, var_list=g_vars)

    return d_train_opt, g_train_opt


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
The algorithm for training a GAN is the following:
1. Generate images using G and random noise (G predicts random images)
2. Perform a Batch update of weights in A given generated images, real images, and labels.
3. Perform a Batch update of weights in G given noise and forced “real” labels in the full GAN.
4. Repeat
Can you write Python code for it?

train_steps=50000
batch_size=256
noise_input = None

for i in range(train_steps):
    images_train = x_train[np.random.randint(0,x_train.shape[0], size=batch_size),:,:,:]
    noise = np.random.normal(0.0, 1.0, size=[batch_size, 100])
    images_fake = Gen.predict(noise)

    make_trainable(Dis, True)
    x = np.concatenate((images_train, images_fake))
    y = np.ones([2*batch_size, 1])
    y[batch_size:, :] = 0
    d_loss = DM.train_on_batch(x, y)

    make_trainable(Dis, False)
    y = np.ones([batch_size, 1])
    noise = np.random.normal(0.0, 1.0, size=[batch_size, 100])
    a_loss = AM.train_on_batch(noise, y)
    Gen.save('Generator_model.h5')


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
These SVHN files are .mat files typically used with Matlab. However, we can load them in with scipy.io.loadmat which we imported above.
Can you write Python code for it?

trainset = loadmat(data_dir + 'train_32x32.mat')
testset = loadmat(data_dir + 'test_32x32.mat')


I am working on the file "DCGAN/DCGAN.ipynb".
The first task is:
When training the GAN we are searching for an equilibrium point, which is the optimal point in a minimax game:
- The generator will models the real data,
- and the discriminator will output probability of 0.5 as the output of the generator = real data
Can you write Python code for it?

# Adversarial model
optimizer = Adam(lr=0.0002, beta_1=0.5)
AM = Sequential()
AM.add(Gen)
AM.add(Dis)
AM.compile(loss='binary_crossentropy', optimizer=optimizer,metrics=['accuracy'])
AM.summary()

# Freeze weights in discriminator D for stacked training
def make_trainable(net, val):
    net.trainable = val
    for l in net.layers:
        l.trainable = val

make_trainable(Dis, False)


I am working on the file "DCGAN/DCGAN_Exercises.ipynb".
The first task is:
And another function we can use to train our network. Notice when we call generator to create the samples to display, we set training to False. That's so the batch normalization layers will use the population statistics rather than the batch statistics. Also notice that we set the net.input_real placeholder when we run the generator's optimizer. The generator doesn't actually use it, but we'd get an errror without it because of the tf.control_dependencies block we created in model_opt.
Can you write Python code for it?


I am working on the file "DCGAN/DCGAN_Exercises.ipynb".
The first task is:
Discriminator
Here you'll build the discriminator. This is basically just a convolutional classifier like you've build before. The input to the discriminator are 32x32x3 tensors/images. You'll want a few convolutional layers, then a fully connected layer for the output. As before, we want a sigmoid output, and you'll need to return the logits as well. For the depths of the convolutional layers I suggest starting with 16, 32, 64 filters in the first layer, then double the depth as you add layers. Note that in the DCGAN paper, they did all the downsampling using only strided convolutional layers with no maxpool layers.
You'll also want to use batch normalization with tf.layers.batch_normalization on each layer except the first convolutional and output layers. Again, each layer should look something like convolution > batch norm > leaky ReLU.
Note: in this project, your batch normalization layers will always use batch statistics. (That is, always set training to True.) That's because we are only interested in using the discriminator to help train the generator. However, if you wanted to use the discriminator for inference later, then you would need to set the training parameter appropriately.

Exercise: Build the convolutional network for the discriminator. The input is a 32x32x3 images, the output is a sigmoid plus the logits. Again, use Leaky ReLU activations and batch normalization on all the layers except the first.
Can you write Python code for it?

def discriminator(x, reuse=False, alpha=0.2):
    with tf.variable_scope('discriminator', reuse=reuse):
        # Input layer is 32x32x3
        x1 = tf.layers.conv2d(x, 64, 5, strides=2, padding='same')
        relu1 = tf.maximum(alpha * x1, x1)

        x2 = tf.layers.conv2d(x1, 128, 5, strides=2, padding='same')
        bn2 = tf.layers.batch_normalization(x1, training=True)
        relu2 = tf.maximum(alpha * bn2, bn2)

        x3 = tf.layers.conv2d(relu2, 256, 5, strides=2, padding='same')
        bn3 = tf.layers.batch_normalization(x3, training=True)
        relu3 = tf.maximum(alpha * bn3, bn3)

        flattened = tf.reshape(relu3, (-1, 4*4*256))
        logits = tf.layers.dense(flattened, 1)
        out = tf.sigmoid(logits)

        return out, logits


I am working on the file "DCGAN/DCGAN_Exercises.ipynb".
The first task is:
Generator
Here you'll build the generator network. The input will be our noise vector z as before. Also as before, the output will be a $tanh$ output, but this time with size 32x32 which is the size of our SVHN images.
What's new here is we'll use convolutional layers to create our new images. The first layer is a fully connected layer which is reshaped into a deep and narrow layer, something like 4x4x1024 as in the original DCGAN paper. Then we use batch normalization and a leaky ReLU activation. Next is a transposed convolution where typically you'd halve the depth and double the width and height of the previous layer. Again, we use batch normalization and leaky ReLU. For each of these layers, the general scheme is convolution > batch norm > leaky ReLU.
You keep stacking layers up like this until you get the final transposed convolution layer with shape 32x32x3. Below is the archicture used in the original DCGAN paper:

Note that the final layer here is 64x64x3, while for our SVHN dataset, we only want it to be 32x32x3.

Exercise: Build the transposed convolutional network for the generator in the function below. Be sure to use leaky ReLUs on all the layers except for the last tanh layer, as well as batch normalization on all the transposed convolutional layers except the last one.
Can you write Python code for it?

def generator(z, output_dim, reuse=False, alpha=0.2, training=True):
    with tf.variable_scope('generator', reuse=reuse):
        # First fully connected layer
        x1 = tf.layers.dense(z, 4*4*512)

        x1 = tf.reshape(-1, 4, 4, 512)
        x1 = tf.layers.batch_normalization(x1, training=training)
        x1 = tf.maximum(alpha * x1, x1)

        x2 = tf.layers.conv2d_transpose(x1, 256, 5, strides=2, padding='same')
        x2 = tf.layers.batch_normalization(x2, training=training)
        x2 = tf.maximum(alpha * x2, x2)

        x3 = tf.layers.conv2d_transpose(x2, 128, 5, strides=2, padding='same')
        x3 = tf.layers.batch_normalization(x3, training=training)
        x3 = tf.maximum(alpha * x3, x3)

        # Output layer, 32x32x3
        logits = tf.layers.conv2d_transpose(x3, output_dim, 5, strides=2, padding='same')

        out = tf.tanh(logits)

        return out


I am working on the file "DCGAN/DCGAN_Exercises.ipynb".
The first task is:
Getting the data
Here you can download the SVHN dataset. Run the cell above and it'll download to your machine.
Can you write Python code for it?

from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm

data_dir = 'data/'

if not isdir(data_dir):
    raise Exception("Data directory doesn't exist!")

class DLProgress(tqdm):
    last_block = 0

    def hook(self, block_num=1, block_size=1, total_size=None):
        self.total = total_size
        self.update((block_num - self.last_block) * block_size)
        self.last_block = block_num

if not isfile(data_dir + "train_32x32.mat"):
    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='SVHN Training Set') as pbar:
        urlretrieve(
            'http://ufldl.stanford.edu/housenumbers/train_32x32.mat',
            data_dir + 'train_32x32.mat',
            pbar.hook)

if not isfile(data_dir + "test_32x32.mat"):
    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='SVHN Training Set') as pbar:
        urlretrieve(
            'http://ufldl.stanford.edu/housenumbers/test_32x32.mat',
            data_dir + 'test_32x32.mat',
            pbar.hook)


I am working on the file "DCGAN/DCGAN_Exercises.ipynb".
The first task is:
Here is a function for displaying generated images.
Can you write Python code for it?

def view_samples(epoch, samples, nrows, ncols, figsize=(5,5)):
    fig, axes = plt.subplots(figsize=figsize, nrows=nrows, ncols=ncols,
                             sharey=True, sharex=True)
    for ax, img in zip(axes.flatten(), samples[epoch]):
        ax.axis('off')
        img = ((img - img.min())*255 / (img.max() - img.min())).astype(np.uint8)
        ax.set_adjustable('box-forced')
        im = ax.imshow(img, aspect='equal')

    plt.subplots_adjust(wspace=0, hspace=0)
    return fig, axes


I am working on the file "DCGAN/DCGAN_Exercises.ipynb".
The first task is:
Here we need to do a bit of preprocessing and getting the images into a form where we can pass batches to the network. First off, we need to rescale the images to a range of -1 to 1, since the output of our generator is also in that range. We also have a set of test and validation images which could be used if we're trying to identify the numbers in the images.
Can you write Python code for it?

def scale(x, feature_range=(-1, 1)):
    # scale to (0, 1)
    x = ((x - x.min())/(255 - x.min()))

    # scale to feature_range
    min, max = feature_range
    x = x * (max - min) + min
    return x

class Dataset:
    def __init__(self, train, test, val_frac=0.5, shuffle=False, scale_func=None):
        split_idx = int(len(test['y'])*(1 - val_frac))
        self.test_x, self.valid_x = test['X'][:,:,:,:split_idx], test['X'][:,:,:,split_idx:]
        self.test_y, self.valid_y = test['y'][:split_idx], test['y'][split_idx:]
        self.train_x, self.train_y = train['X'], train['y']

        self.train_x = np.rollaxis(self.train_x, 3)
        self.valid_x = np.rollaxis(self.valid_x, 3)
        self.test_x = np.rollaxis(self.test_x, 3)

        if scale_func is None:
            self.scaler = scale
        else:
            self.scaler = scale_func
        self.shuffle = shuffle

    def batches(self, batch_size):
        if self.shuffle:
            idx = np.arange(len(dataset.train_x))
            np.random.shuffle(idx)
            self.train_x = self.train_x[idx]
            self.train_y = self.train_y[idx]

        n_batches = len(self.train_y)//batch_size
        for ii in range(0, len(self.train_y), batch_size):
            x = self.train_x[ii:ii+batch_size]
            y = self.train_y[ii:ii+batch_size]

            yield self.scaler(x), self.scaler(y)


I am working on the file "DCGAN/DCGAN_Exercises.ipynb".
The first task is:
Hyperparameters
GANs are very senstive to hyperparameters. A lot of experimentation goes into finding the best hyperparameters such that the generator and discriminator don't overpower each other. Try out your own hyperparameters or read the DCGAN paper to see what worked for them.

Exercise: Find hyperparameters to train this GAN. The values found in the DCGAN paper work well, or you can experiment on your own. In general, you want the discriminator loss to be around 0.3, this means it is correctly classifying images as fake or real about 50% of the time.
Can you write Python code for it?

real_size = (32,32,3)
z_size = 100
learning_rate = 0.001
batch_size = 64
epochs = 1
alpha = 0.01
beta1 = 0.9

# Create the network
net = GAN(real_size, z_size, learning_rate, alpha=alpha, beta1=beta1)

# Load the data and train the network here
dataset = Dataset(trainset, testset)
losses, samples = train(net, dataset, epochs, batch_size, figsize=(10,5))

fig, ax = plt.subplots()
losses = np.array(losses)
plt.plot(losses.T[0], label='Discriminator', alpha=0.5)
plt.plot(losses.T[1], label='Generator', alpha=0.5)
plt.title("Training Losses")
plt.legend()

_ = view_samples(-1, samples, 6, 12, figsize=(10,5))


I am working on the file "DCGAN/DCGAN_Exercises.ipynb".
The first task is:
Model Loss
Calculating the loss like before, nothing new here.
Can you write Python code for it?


I am working on the file "DCGAN/DCGAN_Exercises.ipynb".
The first task is:
These SVHN files are .mat files typically used with Matlab. However, we can load them in with scipy.io.loadmat which we imported above.
Can you write Python code for it?


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Arterial gases only
Can you write Python code for it?

# Select only recordings where at least 10 ARTERIAL blood gases are available
select_recs_art = [key for key in pCO2s if
               len(pCO2s[key].ix[pCO2s[key]['Blood specimen type, POC'] == 'Arteri...']) >= 10]
select_recs_art = sorted(select_recs_art)
select_recs_art;

recordings = select_recs_art[0:1]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    visualise_DCO2_corr_art(recordings, i, xlimoffset = 20, ylimoffset = 5,
                            textboxoffset_x = 0.65, textboxoffset_y = 1.3)

plt.close()

recordings = select_recs_art[1:2]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    visualise_DCO2_corr_art(recordings, i, xlimoffset = 20, ylimoffset = 5,
                            textboxoffset_x = 0.5, textboxoffset_y = 1.2)

plt.close()

recordings = select_recs_art[2:3]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    visualise_DCO2_corr_art(recordings, i, xlimoffset = 20, ylimoffset = 5,
                            textboxoffset_x = 0.45, textboxoffset_y = 1)

plt.close()

recordings = select_recs_art[3:4]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    visualise_DCO2_corr_art(recordings, i, xlimoffset = 20, ylimoffset = 5,
                            textboxoffset_x = 1.3, textboxoffset_y = 1.25)

plt.close()

recordings = select_recs_art[4:5]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
   ...


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Calculate DCO2s  and weight-corrected DCO2s for 10 minute periods before the blood gases, combine them with the pCO2 readings, collect them in a dictionary of dictionaries and write it to multisheet Excel files.
Can you write Python code for it?

def DCO2_calculator(recording, column, time, interval, offset, ):

    '''
    in:
    -recording: recording name (string)
    -column: column to calculate the statistics on, e.g. DCO2 or DCO2_corr
    -time: time of blood gas (string)
    -interval: time in minutes (int)
    -offset: time in minutes (int)

    out:
    tuple of 9 values:
    - number of DCO2 measurements over the period
    - mean
    - st deviation
    - median
    - mad (mean absolute deviation)
    - minimum
    - 25th centile
    - 75th centile
    - maximum

    Calculates statistics about DCO2 values before a blood gas. The interval in 'interval' long (in minutes)
    and it ends 'offset' minutes before the blood gas

    '''

    #time is given in nanoseconds: 1 min = 60000000000 nanoseconds
    end = pd.to_datetime(time) - pd.to_timedelta(offset * 60000000000)

    start = end - pd.to_timedelta(interval * 60000000000)

    # 1 sec (1000000000 nsec) needs to be taken away otherwise the period will be 901 seconds
    # as it includes the last second
    data = slow_measurements_hfov[recording][start : end - pd.to_timedelta(1000000000)]

    stats = stats_calculator(data, column)

    return stats

# Create a dictionary of DCO2 calculations
DCO2_stats_gases = {}
DCO2_stats_gases[interval, offset] = {}
columns = ['number_of_seconds', 'mean', 'stdev', 'median', 'mad', 'min', '25pc', '75pc', 'max', ]

for recording in recordings:
    lst = []
    for gas in gas_list[recording]:
        lst.append(DCO2_calculator(recording,'DCO2', gas, interval, offset))
    stats = DataFrame(lst, columns = columns, index = gas_list[recording])
    DCO2_stats_gases[interval, offset][recording] = stats

    # add pCO2 data
    DCO2_stats_gases[interval, offset][recording] = \
        DCO2_stats_gases[interval, offset][recording].join(pCO2s[recording])

    # Dropping the rows from the DataFrame that do not have DCO2 or pCO2 values
    # Some rows do not have DCO2 values because the baby received conventional...


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Calculate weight-corrected high frequency tidal volumes (VThfs) before the blood gases, combine them with the pCO2 readings, collect them in a dictionary of dictionaries
Can you write Python code for it?

# Create a dictionary of DCO2 calculations
VThf_stats_gases = {}
VThf_stats_gases[interval, offset] = {}
columns = ['number_of_seconds', 'mean', 'stdev', 'median', 'mad', 'min', '25pc', '75pc', 'max', ]

for recording in recordings:
    lst = []
    for gas in gas_list[recording]:
        lst.append(DCO2_calculator(recording,'VThf_kg', gas, interval, offset))
    stats = DataFrame(lst, columns = columns, index = gas_list[recording])
    VThf_stats_gases[interval, offset][recording] = stats

    # add pCO2 data
    VThf_stats_gases[interval, offset][recording] = \
        VThf_stats_gases[interval, offset][recording].join(pCO2s[recording])

    # Dropping the rows from the DataFrame that do not have DCO2 or pCO2 values
    # Some rows do not have DCO2 values because the baby received conventional ventilation
    # during this period (flanked by HFOV periods)
    # Some rows do not have pCO2 data because the blood gas was insufficient
    VThf_stats_gases[interval, offset][recording].dropna(axis = 0, how = 'any', subset = ['mean', 'pCO2, POC']
                                                         , inplace = True)


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Calculate weight-square-corrected VThfs before the blood gases, combine them with the pCO2 readings, collect them in a dictionary of dictionaries
Can you write Python code for it?

# Create a dictionary of DCO2 calculations
VThf_sq_stats_gases = {}
VThf_sq_stats_gases[interval, offset] = {}
columns = ['number_of_seconds', 'mean', 'stdev', 'median', 'mad', 'min', '25pc', '75pc', 'max', ]

for recording in recordings:
    lst = []
    for gas in gas_list[recording]:
        lst.append(DCO2_calculator(recording,'VThf_kg_2', gas, interval, offset))
    stats = DataFrame(lst, columns = columns, index = gas_list[recording])
    VThf_sq_stats_gases[interval, offset][recording] = stats

    # add pCO2 data
    VThf_sq_stats_gases[interval, offset][recording] = \
        VThf_sq_stats_gases[interval, offset][recording].join(pCO2s[recording])

    # Dropping the rows from the DataFrame that do not have DCO2 or pCO2 values
    # Some rows do not have DCO2 values because the baby received conventional ventilation
    # during this period (flanked by HFOV periods)
    # Some rows do not have pCO2 data because the blood gas was insufficient
    VThf_sq_stats_gases[interval, offset][recording].dropna(axis = 0, how = 'any', subset = ['mean', 'pCO2, POC']
                                                         , inplace = True)


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Change the index of PCO2s into single index format
Can you write Python code for it?

# Change the index of pCO2s into single index format

for recording in recordings:
    # print('processing CO2 data for %s' % recording)
    time_list_all = []
    for i in range(len(pCO2s[recording])):
        day = str(pCO2s[recording].index[i][0])[:10]
        time = str(pCO2s[recording].index[i][1])
        date_time = day + ' ' + time
        time_list_all.append(date_time)

    pCO2s[recording].index = time_list_all

# Convert the indices of the pCO2s DataFrames to datetime index

for recording in recordings:
    try:
        pCO2s[recording].index = pd.to_datetime(pCO2s[recording].index)
    except:
        print('%s cannot be converted' % recording)


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Clean up DCO2 data
Can you write Python code for it?

# resampling to adjust the time stamps to full seconds
for recording in recordings:
    slow_measurements[recording] = slow_measurements[recording].resample('1S').mean()

# Remove rows which have no DCO2 data - this keeps HFOV periods only
for recording in recordings:
    try:
        slow_measurements_hfov[recording] = slow_measurements[recording].dropna(subset = ['5001|DCO2 [10*mL^2/s]'])
    except KeyError:
        print('%s cannot be converted' % recording )

for recording in recordings:

    # Create a column in the DataFrame with the DCO2 data. The original values need to be multiplied by 10
    # as in the the downloaded data they are expressed as 1/10th of the DCO2 readings (see original column labels)
    slow_measurements_hfov[recording]['DCO2'] = \
        DataFrame(slow_measurements_hfov[recording]['5001|DCO2 [10*mL^2/s]'] * 10)

    # this normalizes the DCO2 to the body weight. DCO2 = f x VT x VT. If VT is expressed as mL/kg then DCO2_corr
    # should be 1/sec x mL/kg x mL/kg = mL^2 / kg^2 / sec
    slow_measurements_hfov[recording]['DCO2_corr'] = \
        DataFrame(slow_measurements_hfov[recording]['5001|DCO2 [10*mL^2/s]'] * 10 / (current_weights[recording] ** 2))

    # This column normalizes the VThf to body weight
    slow_measurements_hfov[recording]['VThf_kg'] = \
        DataFrame(slow_measurements_hfov[recording]['5001|VThf [mL]']  / current_weights[recording])

    # This column normalizs the VThf to body weight squares
    slow_measurements_hfov[recording]['VThf_kg_2'] = slow_measurements_hfov[recording]['VThf_kg']**2


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Combine all VThf-squared data into single dataframe
Can you write Python code for it?

VThf_sq_stats_gases_all = {}
lst = []
for recording in recordings:
    VThf_sq_stats_gases[interval, offset][recording]['recording'] = recording
    lst.append(VThf_sq_stats_gases[interval, offset][recording])
VThf_sq_stats_gases_all[interval, offset] = pd.concat(lst)


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Combine all VThfs data into single dataframe
Can you write Python code for it?

VThf_stats_gases_all = {}
lst = []
for recording in recordings:
    VThf_stats_gases[interval, offset][recording]['recording'] = recording
    lst.append(VThf_stats_gases[interval, offset][recording])
VThf_stats_gases_all[interval, offset] = pd.concat(lst)


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Combine all leak data into single dataframes (one for each) and write them to excel files
Can you write Python code for it?

DCO2_stats_gases_leak_all = {}

lst = []
for recording in recordings:
    DCO2_stats_gases_leak[interval, offset][recording]['recording'] = recording
    lst.append(DCO2_stats_gases_leak[interval, offset][recording])

DCO2_stats_gases_leak_all[interval, offset] = pd.concat(lst)
DCO2_stats_gases_leak_all[interval, offset].columns = \
    ['DCO2', 'DCO2_corr', 'leak%', 'MV_leak', 'MVi', 'MVe', 'pCO2', 'specimen', 'recording']

# Mean leak more that 5% before the blood gas
len(DCO2_stats_gases_leak_all[10,2][DCO2_stats_gases_leak_all[10,2]['leak%'] > 5]);

# Mean leak more that 10% before the blood gas
len(DCO2_stats_gases_leak_all[10,2][DCO2_stats_gases_leak_all[10,2]['leak%'] > 10]);

# Mean leak more that 25% before the blood gas
len(DCO2_stats_gases_leak_all[10,2][DCO2_stats_gases_leak_all[10,2]['leak%'] > 25]);

# Mean leak more that 50% before the blood gas
len(DCO2_stats_gases_leak_all[10,2][DCO2_stats_gases_leak_all[10,2]['leak%'] > 50]);


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Create DataFrames with leak data together with the mean DCO2, DCO2_corr and with PCO2 data
Can you write Python code for it?

def parameter_calculator(recording, parameter, time, interval, offset, ):

    '''
    in:
    -recording: recording name (string)
    -parameter: parameter to calculate the mean on, e.g. DCO2 or DCO2_corr
    -time: time of blood gas (string)
    -interval: time in minutes (int)
    -offset: time in minutes (int)

    out:
    -

    Calculates the mean value of the parameter before a blood gas.
    The interval in 'interval' long (in minutes) and it ends 'offset' minutes before the blood gas

    '''

    # time is given in nanoseconds: 1 min = 60000000000 nanoseconds
    end = pd.to_datetime(time) - pd.to_timedelta(offset * 60000000000)

    start = end - pd.to_timedelta(interval * 60000000000)

    # 1 sec (1000000000 nsec) needs to be taken away otherwise the period will be 901 seconds
    # as it includes the last second
    data = slow_measurements_hfov[recording][start : end - pd.to_timedelta(1000000000)]

    return data[parameter].mean()

# Create a dictionary of Vthf, DCO2 calculations together with the leak data
DCO2_stats_gases_leak = {}
DCO2_stats_gases_leak[interval, offset] = {}

for recording in recordings:
    columns = ['DCO2', 'DCO2_corr', '5001|% leak [%]', '5001|MVleak [L/min]', '5001|MVi [L/min]',
               '5001|MVe [L/min]']
    par_dict = {}
    for gas in gas_list[recording]:
        lst = []
        lst.append(parameter_calculator(recording,'DCO2', gas, interval, offset))
        lst.append(parameter_calculator(recording,'DCO2_corr', gas, interval, offset))
        lst.append(parameter_calculator(recording,'5001|% leak [%]', gas, interval, offset))
        lst.append(parameter_calculator(recording,'5001|MVleak [L/min]', gas, interval, offset))
        lst.append(parameter_calculator(recording,'5001|MVi [L/min]', gas, interval, offset))
        lst.append(parameter_calculator(recording,'5001|MVe [L/min]', gas, interval, offset))
        par_dict[gas] = lst

    DCO2_stats_gases_leak[interval, offset][recording] = \
       ...


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Create ROC curve
Can you write Python code for it?

val = range(0, 180, 5)
true_pos = []
false_pos = []
true_neg = []
false_neg = []

a = DCO2_stats_gases_leak_all[10,2]

for i in val:
    pos = a[a['DCO2_corr'] >= i]
    neg = a[a['DCO2_corr'] < i]
    tp = len(pos[pos['pCO2'] <= 8])
    fp = len(pos[pos['pCO2'] > 8])
    tn = len(neg[neg['pCO2'] > 8])
    fn = len(neg[neg['pCO2'] <= 8])
    true_pos.append(tp)
    false_pos.append(fp)
    true_neg.append(tn)
    false_neg.append(fn)

DCO2_test = DataFrame({'tp': true_pos, 'fp': false_pos, 'tn': true_neg, 'fn': false_neg}, index = val )

DCO2_test['sensitivity'] = round(DCO2_test.tp / (DCO2_test.tp + DCO2_test.fn), 3)
DCO2_test['specificity'] = round(DCO2_test.tn / (DCO2_test.tn + DCO2_test.fp), 3)
DCO2_test['pos_pred_value'] = round(DCO2_test.tp / (DCO2_test.tp + DCO2_test.fp), 3)
DCO2_test['neg_pred_value'] = round(DCO2_test.tn / (DCO2_test.tn + DCO2_test.fn), 3)
DCO2_test['Youden'] = round(DCO2_test.sensitivity + DCO2_test.specificity - 1, 3)

DCO2_test[DCO2_test.Youden == DCO2_test.Youden.max()];

DCO2_test['1-specificity'] = 1 - DCO2_test['specificity']

DCO2_test.sort_values('1-specificity', inplace = True)

DCO2_test;

DCO2_test['1-specificity'].iloc[30];

DCO2_test['sensitivity'].iloc[30];

AUC = 0 # Area under the ROC curve

for i in range(len(DCO2_test['1-specificity'])-2):
    c_high = ((DCO2_test['1-specificity'].iloc[i+1] - DCO2_test['1-specificity'].iloc[i]) *
        DCO2_test['sensitivity'].iloc[i+1])
    c_low = ((DCO2_test['1-specificity'].iloc[i+1] - DCO2_test['1-specificity'].iloc[i]) *
        DCO2_test['sensitivity'].iloc[i])
    c = (c_high + c_low) / 2
    AUC += c


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Create a dictionary containing the time of  blood gases taken during HFOV recording
Can you write Python code for it?


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Define function to import data
Can you write Python code for it?

def data_loader(lst):

    '''
    lst ->  DataFrame

    - Takes a list of csv files (given as filenames with absolute paths) and import them as a dataframes
    - Combines the 'Date' and 'Time' columns into a Datetime index while keeping the original columns
    - If there are more files it concatenates them into a single dataframe.
    - It returns the concatenated data as one DataFrame.

    '''

    data  = []
    for i, filename in enumerate(lst):
        data.append(pd.read_csv(lst[i], keep_date_col = 'True', parse_dates = [['Date', 'Time']]))
    data = pd.concat(data)
    data.index = data.Date_Time
    return data


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Define functions to visualize individual recordings and to save them to files
Can you write Python code for it?

def visualise_DCO2_corr(rec, i, xlimoffset = 20, ylimoffset = 2, textboxoffset_x = 0.55, textboxoffset_y = 0.9):

    '''
    input:
    - rec = list of recordings (list)
    - xlimoffset = how much longer is the X-axis than the highest X value (int)
    - ylimoffset = how much longer is the y-axis than the highest y value (int)
    - textboxoffset_x = vertical position of the textbox compared to X-lim, fraction (float)
    - textboxoffset_y = horizontal position of the textbox compared to y-lim, fraction (float)

    Draws a pCO2-DCO2_corr graph using data in recordings 'rec'. Points belonging to different recordings
    have different markers and colors.
    A correlation line is also drawn.
    It also calculates Spearman's correlation coefficient with confidence intervals and p-value.
    Puts these data on the graph.
    '''

    total = DCO2_stats_gases_all[interval, offset]
    sample = total.ix[total['recording'] == recording]
    x = sample['mean']
    y = sample['pCO2, POC']
    if len(x) == 0:
        return

    ax = fig.add_subplot(len(rec),1,i+1);
    plt.scatter(x , y, color = 'black', marker = markers[0], s = 60)
    plt.ylabel(r'pCO$ \rm _2$ (kPa)', fontsize  = 14)
    plt.xlabel(r'DCO$ \rm _2$_corr (mL$\rm^2$/sec/kg$^2$)', fontsize  = 14)

    plt.title(r'%s: Weight-corrected DCO$ \rm _2$' % recording , fontsize = 14)
    xlim = ax.get_xlim()[1]
    ylim = ax.get_ylim()[1]
    plt.xlim([0, xlim + xlimoffset])
    plt.ylim([0, ylim + ylimoffset])
    # Polynomial Coefficients
    coeffs = np.polyfit(x, y, deg = 1)
    result = coeffs.tolist()

    # Fit a trendline
    l = np.poly1d(coeffs)
    plt.plot(x,l(x),'r--')

    # Calculate pearson's correlation coefficient with confidence intervals, coefficiet of determination and p value
    r , lcl, ucl , r2, p = correl(x,y)
    p = round(p, 4)

    # print the equation on the graph area

    text = 'y=%.4fx+(%.4f)\nr=%.4f (%.4f , %.4f)\np=%.4f' % (result[0], result[1], r, lcl, ucl, p)

    plt.text(xlim *...


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Figure 2
Can you write Python code for it?

x = DCO2_stats_gases_all[interval, offset]['mean']
y = DCO2_corr_stats_gases_all[interval, offset]['mean']

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)
ax1 = fig.add_subplot(1, 1, 1);
bp = plt.boxplot([x, y] )

plt.setp(bp['boxes'], color='black')
plt.setp(bp['whiskers'], color='black')
plt.setp(bp['medians'], color='red')
plt.setp(bp['fliers'], color='black', marker='+')
plt.ylabel("Units", fontsize  = 14)
plt.xlabel("", fontsize  = 14)
plt.xticks([1,2], [r'DCO$ \rm _2$ (mL$\rm^2$/sec)', r'DCO$ \rm _2$ (mL$\rm^2$/sec/kg$^2$)'], size = 14)
plt.tick_params(axis='both', which='major', labelsize=14)
plt.title(r'Boxplots of all DCO$ \rm _2$ data' , fontsize = 14);

# calculate p values
t_stats, p = ttest_rel(x, y)
ax1.text(ax1.get_xlim()[1]-1.2, 550, 'p<0.001', color = 'red', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'None', 'alpha':1, 'pad':10});

fig.savefig('%s/Figure_2_color.tiff' % dir_write, dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Figure 3
Can you write Python code for it?

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    total = DCO2_corr_stats_gases_all[interval, offset]
    sample = total.ix[total['recording'] == recording]
    x = sample['mean']
    y = sample['pCO2, POC']
    fig.add_subplot(1,1,1);
    ax = plt.scatter(x , y, color = colors[i], marker = markers[i], s = 25)

plt.axvline(x= 50 , linewidth=1, color = 'black')
plt.axhline(y= 8,  linewidth=1, color = 'black')
plt.ylabel(r'pCO$ \rm _2$ (kPa)', fontsize  = 14)
plt.xlabel(r'DCO$ \rm _2$_corr (mL$\rm^2$/sec/kg$^2$) - all gases', fontsize  = 14)
plt.tick_params(axis='both', which='major', labelsize=14)

plt.xlim([0,200])
plt.title(r'Weight-corrected DCO$ \rm _2$ - all gases' , fontsize = 14)

a = DCO2_corr_stats_gases_all[interval, offset]['mean']
b = DCO2_corr_stats_gases_all[interval, offset]['pCO2, POC']

# Polynomial Coefficients
coeffs = np.polyfit(a, b, deg = 1)
result = coeffs.tolist()

# Fit a trendline
l = np.poly1d(coeffs)
plt.plot(a,l(a), '--', color = 'red', linewidth = 2)

# Calculate pearson's correlation coefficient with confidence intervals, coefficiet of determination and p value
r , lcl, ucl , r2, p = correl(a,b)

# print the equation on the graph area
text = 'y=%.4fx+(%.4f)\nr=%.4f (%.4f , %.4f)\np<0.001' % (result[0], result[1], r, lcl, ucl)
plt.text(a.max()*0.47, b.max() * 0.85, text, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'black', 'alpha':1, 'pad':10})

fig.savefig('%s/Figure_3_color.tiff' % dir_write, dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Figure 4
Can you write Python code for it?

# ROC curve
# A corrected DCO2 > 50 ml/min/kg2 predict a pCO2 < 8 kPa with a senstiivty of 0.390 and a specificity of 0.825
# (Youden score = 0.215)
x = [0, 1]; y = [0, 1]

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
fig.add_subplot(1,1,1)

text = r'DCO$ \rm _2$corr = 50'
text2 = 'AUC = %.3f' % AUC

plt.plot(1 - DCO2_test.specificity , DCO2_test.sensitivity, c = 'black')
ax = plt.plot(x, y, c = 'black', linestyle = '--', )
plt.vlines(1 - 0.824561, 1 - 0.824561, 0.390, color='k', linewidth = 3)
plt.text(0.05, 0.60, text, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'black', 'alpha':1, 'pad':10});

plt.text(0.6, 0.3, text2, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'black', 'alpha':1, 'pad':10});

plt.title('ROC curve', size = 14)
plt.ylabel('Sensitivity', fontsize  = 14)
plt.xlabel('1 - Specificity', fontsize  = 14)
plt.grid()

fig.savefig('%s/Figure_4.tiff' % dir_write , dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Figure 5
Can you write Python code for it?

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.7)

subset = DCO2_stats_gases_leak_all[10,2][DCO2_stats_gases_leak_all[10,2]['leak%'] < 10]
for i, recording in enumerate(recordings):
    total = subset
    sample = total.ix[total['recording'] == recording]
    x = sample['DCO2_corr']
    y = sample['pCO2']
    fig.add_subplot(1,1,1);
    plt.scatter(x , y, color = colors[i], marker = markers[i], s = 25)

plt.ylabel(r'pCO$ \rm _2$ (kPa)', fontsize  = 14)
plt.xlabel(r'DCO$ \rm _2$_corr (mL$\rm^2$/sec/kg$^2$)', fontsize  = 14)
plt.xlim([0,200])
plt.title(r'Weight-corrected DCO$ \rm _2$ - leak < 10%' , fontsize = 14)

a = total['DCO2_corr']
b = total['pCO2']

# Polynomial Coefficients
coeffs = np.polyfit(a, b, deg = 1)
result = coeffs.tolist()

# Fit a trendline
l = np.poly1d(coeffs)
plt.plot(a,l(a),'r--')

# Calculate pearson's correlation coefficient with confidence intervals, coefficiet of determination and p value
r , lcl, ucl , r2, p = correl(a,b)

# print the equation on the graph area
text = 'y=%.4fx+(%.4f)\nr=%.4f (%.4f , %.4f)\np<0.001' % (result[0], result[1], r, lcl, ucl)
plt.text(a.max()*0.52, b.max()*0.84, text, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'red', 'alpha':1, 'pad':10})

fig.savefig('%s/Figure_5_color.tiff' % dir_write, dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Figures in the main article
Figure 1
Can you write Python code for it?

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    total = DCO2_stats_gases_all[interval, offset]
    sample = total.ix[total['recording'] == recording]
    x = sample['mean']
    y = sample['pCO2, POC']
    fig.add_subplot(1,1,1);
    plt.scatter(x , y, color = colors[i] , marker = markers[i], s = 25)
    plt.ylabel(r'pCO$ \rm _2$ (kPa)', fontsize  = 14)
    plt.xlabel(r'DCO$ \rm _2$ (mL$\rm^2$/sec)', fontsize  = 14)
    plt.tick_params(axis='both', which='major', labelsize=14)
    plt.xlim([0,600])
    plt.title(r'Uncorrected DCO$ \rm _2$ - all gases' , fontsize = 14);

a = DCO2_stats_gases_all[interval, offset]['mean']
b = DCO2_stats_gases_all[interval, offset]['pCO2, POC']

# Polynomial Coefficients
coeffs = np.polyfit(a, b, deg = 1)
result = coeffs.tolist()

# Fit a trendline
l = np.poly1d(coeffs)
plt.plot(a,l(a),'--', color = 'red')

# Calculate pearson's correlation coefficient with confidence intervals, coefficiet of determination and p value
r , lcl, ucl , r2, p = correl(a,b)

# print the equation on the graph area

text = 'y=%.4fx+(%.4f)\nr=%.4f (%.4f , %.4f)' % (result[0], result[1], r, lcl, ucl)
plt.text(275, b.max()-2, text, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'black', 'alpha':1, 'pad':10});

fig.savefig('%s/Figure_1_color.tiff' % dir_write, dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
For Supplementary Figure 3 of the paper please see the functions creating the individual graphs above.
Supplementary Figure 4
Can you write Python code for it?

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
fig.suptitle('E-Figure 4', size = 14)

subset = DCO2_stats_gases_leak_all[10,2][DCO2_stats_gases_leak_all[10,2]['leak%'] < 10]
for i, recording in enumerate(recordings):
    total = subset
    sample = total.ix[total['recording'] == recording]
    sample_art = sample.ix[subset['specimen'] == 'Arteri...']
    x = sample_art['DCO2_corr']
    y = sample_art['pCO2']
    fig.add_subplot(1,1,1);
    plt.scatter(x , y, color = colors[i], marker = markers[i], s = 25)

plt.ylabel(r'pCO$ \rm _2$ (kPa)', fontsize  = 14)
plt.xlabel(r'DCO$ \rm _2$_corr (mL$\rm^2$/sec/kg$^2$)', fontsize  = 14)
plt.xlim([0,200])
plt.title(r'Weight-corrected DCO$ \rm _2$ - leak<10%' , fontsize = 14)

sample2 = total.ix[total['specimen'] == 'Arteri...']
a = sample2['DCO2_corr']
b = sample2['pCO2']

# Polynomial Coefficients
coeffs = np.polyfit(a, b, deg = 1)
result = coeffs.tolist()

# Fit a trendline
l = np.poly1d(coeffs)
plt.plot(a,l(a),'--', color = 'red')

# Calculate pearson's correlation coefficient with confidence intervals, coefficiet of determination and p value
r , lcl, ucl , r2, p = correl(a,b)

# print the equation on the graph area
text = 'y=%.4fx+(%.4f)\nr=%.4f (%.4f , %.4f)\np=%.3f' % (result[0], result[1], r, lcl, ucl, p)
plt.text(a.max()*0.52, b.max()*0.84, text, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'red', 'alpha':1, 'pad':10})

fig.savefig('%s/E_figure_4_color.tiff' % dir_write, dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Initialize directories, analysis settings, and visualization markers
Can you write Python code for it?

clinical_details = {}
current_weights = {}
blood_gases = {}
pCO2s = {}
sedation = {}
vent_modes = {}
vent_modes_selected = {} # only important mode parameters are kept in this one
vent_settings = {}
vent_settings_selected = {} # only important mode parameters are kept in this one
slow_measurements = {} # 'slow measurements' are the ventilator parameters obtained with 1/sec frequeency
slow_measurements_hfov = {} # this only contains those parts of 'slow measurements' which do have a DCO2 reading
                            # and therefore are hfov ventilation periods

# Analysing DCO2s over "interval" minutes stopping "offset" minutes prior to the time of the blood gas

interval = 10 # 10 minutes
offset = 2 # 2 minutes

# color and marker set for visualization

colors = ['b', 'g', 'r', 'c', 'm', 'y', 'k', 'magenta', 'maroon', 'navy', 'salmon', 'silver',  'tomato',
          'violet', 'orchid', 'peru', 'wheat', 'lime', 'limegreen']
markers = [".", "+", ",", "x", "o", "D", "d", "8", "s", "p", "*", "h", 0, 4, "<", "3",
           1, 5, ">", "4", 2, 6, "^", "2", 3, 7, "v", "1"]


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
List and set the working directory and the directory to write out data
Can you write Python code for it?

# Directory to read the data from
%cd /Users/guszti/ventilation_data
cwd = os.getcwd()

# Directory to write the data to
dir_write = '/Users/guszti/ventilation_data/Analyses/DCO2/revision'


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Supplementary Figure 2
Can you write Python code for it?

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
fig.suptitle('E-Figure 2', size = 14)

for i, recording in enumerate(recordings):
    total = DCO2_corr_stats_gases_all[interval, offset]
    sample = total.ix[total['recording'] == recording]
    sample_art = sample.ix[sample['Blood specimen type, POC'] == 'Arteri...']
    x = sample_art['mean']
    y = sample_art['pCO2, POC']
    fig.add_subplot(1,1,1);
    plt.scatter(x , y, color = colors[i], marker = markers[i], s = 25)

plt.axvline(x= 60 , linewidth=1, color = 'black')
plt.axhline(y= 8,  linewidth=1, color = 'black')
plt.ylabel(r'pCO$ \rm _2$ (kPa)', fontsize  = 14)
plt.xlabel(r'DCO$ \rm _2$_corr (mL$\rm^2$/sec/kg$^2$)', fontsize  = 14)
plt.tick_params(axis='both', which='major', labelsize=14)
plt.xlim([0,200])
plt.title(r'Weight-corrected DCO$ \rm _2$ - arterial gases only' , fontsize = 14)

sample2 = total.ix[total['Blood specimen type, POC'] == 'Arteri...']
a = sample2['mean']
b = sample2['pCO2, POC']

# Polynomial Coefficients
coeffs = np.polyfit(a, b, deg = 1)
result = coeffs.tolist()

# Fit a trendline
l = np.poly1d(coeffs)
plt.plot(a,l(a), '--', color = 'red',  linewidth = 2)

# Calculate pearson's correlation coefficient with confidence intervals, coefficient of determination and p value
r , lcl, ucl , r2, p = correl(a,b)

# print the equation on the graph area

text = 'y=%.4fx+(%.4f)\nr=%.4f (%.4f , %.4f)\np=%.3f' % (result[0], result[1], r, lcl, ucl, p)

plt.text(a.max()*0.48, b.max()*0.83, text, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'red', 'alpha':1, 'pad':10});

fig.savefig('%s/E_figure_2_color.tiff' % dir_write, dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Supplementary Figure 6
Can you write Python code for it?

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
fig.suptitle('E-Figure 6', size = 14)

for i, recording in enumerate(recordings):
    total = VThf_stats_gases_all[interval, offset]
    sample = total.ix[total['recording'] == recording]
    x = sample['mean']
    y = sample['pCO2, POC']
    fig.add_subplot(1,1,1);
    plt.scatter(x , y, color = colors[i], marker = markers[i], s = 25)

plt.ylabel(r'pCO$ \rm _2$ (kPa)', fontsize  = 14)
plt.xlabel(r'VThf (mL/kg)', fontsize  = 14)
plt.xlim([0,5])
# plt.title(r'Tidal volume' , fontsize = 14)

a = VThf_stats_gases_all[interval, offset]['mean']
b = VThf_stats_gases_all[interval, offset]['pCO2, POC']

# Polynomial Coefficients
coeffs = np.polyfit(a, b, deg = 1)
result = coeffs.tolist()

# Fit a trendline
l = np.poly1d(coeffs)
plt.plot(a,l(a),'--', color = 'red')

# Calculate pearson's correlation coefficient with confidence intervals, coefficiet of determination and p value
r , lcl, ucl , r2, p = correl(a,b)

# print the equation on the graph area
text = 'y=%.4fx+(%.4f)\nr=%.4f (%.4f , %.4f)\np<0.001' % (result[0], result[1], r, lcl, ucl)
plt.text(a.max()*0.52, b.max()*0.84, text, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'red', 'alpha':1, 'pad':10})

fig.savefig('%s/E_figure_6_color.tiff' % dir_write, dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Supplementary Figure 7
Can you write Python code for it?

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
fig.suptitle('E-Figure 7', size = 14)

for i, recording in enumerate(recordings):
    total = VThf_sq_stats_gases_all[interval, offset]
    sample = total.ix[total['recording'] == recording]
    x = sample['mean']
    y = sample['pCO2, POC']
    fig.add_subplot(1,1,1);
    plt.scatter(x , y, color = colors[i], marker = markers[i], s = 25)

plt.ylabel(r'pCO$ \rm _2$ (kPa)', fontsize  = 14)
plt.xlabel(r'VThf (mL$^2$/kg$^2$)', fontsize  = 14)
plt.xlabel(r'VThf$^2$ (mL$^2$/kg$^2$)', fontsize  = 14)

a = VThf_sq_stats_gases_all[interval, offset]['mean']
b = VThf_sq_stats_gases_all[interval, offset]['pCO2, POC']

# Polynomial Coefficients
coeffs = np.polyfit(a, b, deg = 1)
result = coeffs.tolist()

# Fit a trendline
l = np.poly1d(coeffs)
plt.plot(a,l(a),'--', color = 'red')

# Calculate pearson's correlation coefficient with confidence intervals, coefficiet of determination and p value
r , lcl, ucl , r2, p = correl(a,b)

# print the equation on the graph area
text = 'y=%.4fx+(%.4f)\nr=%.4f (%.4f , %.4f)\np<0.001' % (result[0], result[1], r, lcl, ucl)
plt.text(a.max()*0.43, b.max()*0.83, text, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'red', 'alpha':1, 'pad':10})

fig.savefig('%s/E_Figure_7_color.tiff' % dir_write, dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Supplementary figures
Supplementary Figure 1
Can you write Python code for it?

fig = plt.figure()
fig.set_size_inches(7, 5)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)
fig.suptitle('E-Figure 1', size = 14)

for i, recording in enumerate(recordings):
    total = DCO2_stats_gases_all[interval, offset]
    sample = total.ix[total['recording'] == recording]
    sample_art = sample.ix[sample['Blood specimen type, POC'] == 'Arteri...']
    x = sample_art['mean']
    y = sample_art['pCO2, POC']
    fig.add_subplot(1,1,1);
    plt.scatter(x , y, color = colors[i], marker = markers[i], s = 25)
    plt.ylabel(r'pCO$ \rm _2$ (kPa)', fontsize  = 14)
    plt.xlabel(r'DCO$ \rm _2$ (mL$\rm^2$/sec)', fontsize  = 14)
    plt.tick_params(axis='both', which='major', labelsize=14)
    plt.xlim([0,600])
    plt.title(r'Uncorrected DCO$ \rm _2$ - arterial gases only' , fontsize = 14);

sample2 = total.ix[total['Blood specimen type, POC'] == 'Arteri...']
a = sample2['mean']
b = sample2['pCO2, POC']

# Polynomial Coefficients
coeffs = np.polyfit(a, b, deg = 1)
result = coeffs.tolist()

# Fit a trendline
l = np.poly1d(coeffs)
plt.plot(a,l(a),'--', color = 'red')

# Calculate pearson's correlation coefficient with confidence intervals, coefficiet of determination and p value
r , lcl, ucl , r2, p = correl(a,b)

# print the equation on the graph area
text = 'y=%.4fx+(%.4f)\nr=%.4f (%.4f , %.4f)' % (result[0], result[1], r, lcl, ucl)
plt.text(a.max()*0.48, b.max()-2, text, color = 'black', style='normal', fontsize=14,
         bbox={'facecolor':'white', 'edgecolor':'black', 'alpha':1, 'pad':10});

fig.savefig('%s/E_Figure_1_color.tiff' % dir_write, dpi = 300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format = 'tiff',
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

plt.close()


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Visualise data from individual recordings where at least ten data points are available
All blood gases
These graphs are shown in Supplementary Figure 3 of the paper.
Can you write Python code for it?

# Select only recordings where at least 10 blood gases are available
select_recs = [key for key in pCO2s if len(pCO2s[key]) >= 10]
select_recs = sorted(select_recs)
select_recs;

recordings = select_recs[0:1]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    visualise_DCO2_corr(recordings, i, xlimoffset = 20, ylimoffset = 6, textboxoffset_x = 0.50, textboxoffset_y = 1.1)

plt.close()

recordings = select_recs[1:2]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    visualise_DCO2_corr(recordings, i, xlimoffset = 20, ylimoffset = 4, textboxoffset_x = 0.66, textboxoffset_y = 1.2)

plt.close()

recordings = select_recs[2:3]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    visualise_DCO2_corr(recordings, i, xlimoffset = 20, ylimoffset = 5, textboxoffset_x = 0.55, textboxoffset_y = 1.2)

plt.close()

recordings = select_recs[3:4]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    visualise_DCO2_corr(recordings, i, xlimoffset = 20, ylimoffset = 5, textboxoffset_x = 0.5, textboxoffset_y = 1.1)

plt.close()

recordings = select_recs[4:5]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)

for i, recording in enumerate(recordings):
    visualise_DCO2_corr(recordings, i, xlimoffset = 20, ylimoffset = 5, textboxoffset_x = 0.55, textboxoffset_y = 1.2)

plt.close()

recordings = select_recs[5:6]

fig = plt.figure()
fig.set_size_inches(8,6)
fig.subplots_adjust(left=None, bottom=None,...


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Write cumulative data to excel files and binary pickle files
Can you write Python code for it?

writer = pd.ExcelWriter('%s/%s_%d_%d.xlsx' % (dir_write, 'DCO2_stats_all_gases', interval, offset) )
DCO2_stats_gases_all[interval, offset].to_excel(writer, 'DCO2')
writer.save()

writer = pd.ExcelWriter('%s/%s_%d_%d.xlsx' % (dir_write, 'DCO2_corr_stats_all_gases', interval, offset) )
DCO2_corr_stats_gases_all[interval, offset].to_excel(writer, 'DCO2_corr')
writer.save()

DCO2_stats_gases_all[interval, offset].to_pickle('%s/%s_%d_%d.p'
                                                 % (dir_write, 'DCO2_stats_all_gases', interval, offset))


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
Write out and import processed HFOV data to binary "pickle" file
Can you write Python code for it?

for recording in recordings:
    print('%s is being pickled' % recording)
    slow_measurements_hfov[recording].to_pickle('%s/%s_slow_measurements_hfov.p' % (dir_write, recording))

# importing from pickles
for recording in recordings:
    slow_measurements_hfov[recording] = pd.read_pickle('%s/%s_slow_measurements_hfov.p' % (dir_write, recording))


I am working on the file "DCO2corr_Github.ipynb".
The first task is:
weight_corrected DCO2
Can you write Python code for it?

DCO2_corr_stats_gases_all = {}

lst = []
for recording in recordings:
    DCO2_corr_stats_gases[interval, offset][recording]['recording'] = recording
    lst.append(DCO2_corr_stats_gases[interval, offset][recording])
DCO2_corr_stats_gases_all[interval, offset] = pd.concat(lst)

DCO2_corr_stats_gases_all[interval, offset];


I am working on the file "DCT/DCT fundamentals.ipynb".
The first task is:
Notice that the first basis $(k = 0)$ does not oscillate, this basis is called the DC. The other basis oscillate and are referred to as AC. This is reminescent of direct current and alternating current:

Notice that oscillations of the basis functions increase as $k$ increase, we will come back to this later. For now, let's use our basis function to perform the DCT on $v$.
Computing the DCT
To compute our ouput vector we need to sum the multiplication of each value of the vector by the corresponding basis.
Can you write Python code for it?

def dct(v):
    N = len(v)
    X = np.zeros([4,1])
    for k in range(0, N):
        X[k] = sum(v[k] * basis(k, N))
    return X

X = dct(v)

print("Transformed input vector %s", X)

pylab.figure(figsize=(15,3))
def showTransformedEnergy(e, title):
    x = np.arange(4)
    pylab.bar(x, e)
    pylab.xticks([1, 2, 3, 4])
    pylab.xlim(0, 4);
    pylab.ylim(-1, 17);
    pylab.title(title)

pylab.subplot(1,2,1)
showTransformedEnergy(X, "Energy of the transformed input vector")
pylab.subplot(1,2,2)
showTransformedEnergy(v, "Energy of the input vector")


I am working on the file "DD_Net_demo.ipynb".
The first task is:
Define data processing functions
Can you write Python code for it?

# Temple resizing function
def zoom(p,target_l=64,joints_num=25,joints_dim=3):
    l = p.shape[0]
    p_new = np.empty([target_l,joints_num,joints_dim])
    for m in range(joints_num):
        for n in range(joints_dim):
            p_new[:,m,n] = medfilt(p_new[:,m,n],3)
            p_new[:,m,n] = inter.zoom(p[:,m,n],target_l/l)[:target_l]
    return p_new

# Calculate JCD feature
def norm_scale(x):
    return (x-np.mean(x))/np.mean(x)

def get_CG(p,C):
    M = []
    iu = np.triu_indices(C.joint_n,1,C.joint_n)
    for f in range(C.frame_l):
        d_m = cdist(p[f],p[f],'euclidean')
        d_m = d_m[iu]
        M.append(d_m)
    M = np.stack(M)
    M = norm_scale(M)
    return M

# Genrate dataset
def data_generator(T,C,le):
    X_0 = []
    X_1 = []
    X_2 = []
    Y = []
    limb_locs = [(0,2),(4,0),(3,0),(8,4),(7,3),(12,8),(11,7),(1,0),(6,1),(5,1),(10,6),(9,5),(14,10),(13,9)]
    print('len_t_pose:',len(T['pose']))
    for i in tqdm(range(len(T['pose']))):
        p = np.copy(T['pose'][i])
        q = np.copy(p)
        q = np.pad(q,((0,0),(0,0),(0,1)), constant_values=0)
        qc = np.zeros((q.shape[0],q.shape[1]-1,q.shape[2]))
        normal_vec_pos = 0
        for j1,j2 in limb_locs:
            joint1_section = np.copy(q[:,j1,:])
            joint2_section = np.copy(q[:,j2,:])
            limb_mat = joint1_section - joint2_section
            obs_vec_mat = np.copy(joint1_section)
            #print('obs_vec_mat.size:',obs_vec_mat.size())
            #print('opx.size:',self.opx.size())
            obs_vec_mat[:,0] = obs_vec_mat[:,0] - 0.023900129681375948
            obs_vec_mat[:,1] = obs_vec_mat[:,1] - 0.37827336942214096
            obs_vec_mat[:,2] = obs_vec_mat[:,2] - 1
            qc[:,normal_vec_pos,:] = np.cross(limb_mat, obs_vec_mat, axis=1)
            normal_vec_pos += 1
        if i == 0:
            print('p.shape:',p.shape)
            print('q.shape:',q.shape)
        p = zoom(p,target_l=C.frame_l,joints_num=C.joint_n,joints_dim=C.joint_d)
        qcz...


I am working on the file "DD_Net_demo.ipynb".
The first task is:
Define network
Can you write Python code for it?

def poses_diff(x):
    H, W = x.get_shape()[1],x.get_shape()[2]
    x = tf.subtract(x[:,1:,...],x[:,:-1,...])
    x = tf.compat.v1.image.resize_nearest_neighbor(x,size=[H,W],align_corners=False) # should not alignment here
    return x

def pose_motion(P,frame_l):
    #print('pose_motion: P.shape:',P.shape)
    P_diff_slow = Lambda(lambda x: poses_diff(x))(P)
    #print('pose_motion: P_diff_slow.shape:',P_diff_slow.shape)
    P_diff_slow = Reshape((frame_l,-1))(P_diff_slow)
    #print('pose_motion: P_diff_slow1.shape:',P_diff_slow.shape)
    P_fast = Lambda(lambda x: x[:,::2,...])(P)
    #print('pose_motion: P_fast.shape:',P_fast.shape)
    P_diff_fast = Lambda(lambda x: poses_diff(x))(P_fast)
    #print('pose_motion: P_diff_fast.shape:',P_diff_fast.shape)
    P_diff_fast = Reshape((int(frame_l/2),-1))(P_diff_fast)
    #print('pose_motion: P_diff_fast1.shape:',P_diff_fast.shape)
    return P_diff_slow,P_diff_fast

def c1D(x,filters,kernel):
    #print('c1D - x.shape:',x.shape)
    x = Conv1D(filters, kernel_size=kernel,padding='same',use_bias=False)(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(alpha=0.2)(x)
    return x

def block(x,filters):
    x = c1D(x,filters,3)
    x = c1D(x,filters,3)
    return x

def d1D(x,filters):
    x = Dense(filters,use_bias=False)(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(alpha=0.2)(x)
    return x

def build_FM(frame_l=32,joint_n=22,joint_d=2,feat_d=231,filters=16):
    M = Input(shape=(frame_l,feat_d))
    P = Input(shape=(frame_l,joint_n,joint_d))
    Q = Input(shape=(frame_l,14,3))

    diff_slow,diff_fast = pose_motion(P,frame_l)
    #print('M.s:',M.shape)

    x = c1D(M,filters*2,1)
    x = SpatialDropout1D(0.1)(x)
    x = c1D(x,filters,3)
    x = SpatialDropout1D(0.1)(x)
    x = c1D(x,filters,1)
    x = MaxPooling1D(2)(x)
    x = SpatialDropout1D(0.1)(x)

    #print('P.s:',P.shape)
    #print('diff_slow.s:',diff_slow.shape)

    x_d_slow = c1D(diff_slow,filters*2,1)
    x_d_slow = SpatialDropout1D(0.1)(x_d_slow)
    x_d_slow =...


I am working on the file "DD_Net_demo.ipynb".
The first task is:
Start train on split 1
Can you write Python code for it?

import keras
lr = 1e-3
DD_Net.compile(loss="categorical_crossentropy",optimizer=adam(lr),metrics=['accuracy'])
lrScheduler = keras.callbacks.ReduceLROnPlateau(monitor='loss', factor=0.5, patience=5, cooldown=5, min_lr=5e-6)
history = DD_Net.fit([X_0,X_1,X_2],Y,
                    batch_size=len(Y),
                    epochs=600,
                    verbose=True,
                    shuffle=True,
                    callbacks=[lrScheduler],
                    validation_data=([X_test_0,X_test_1,X_test_2],Y_test)
                    )
lr = 1e-4
DD_Net.compile(loss="categorical_crossentropy",optimizer=adam(lr),metrics=['accuracy'])
lrScheduler = keras.callbacks.ReduceLROnPlateau(monitor='loss', factor=0.5, patience=5, cooldown=5, min_lr=5e-6)
history = DD_Net.fit([X_0,X_1,X_2],Y,
                    batch_size=len(Y),
                    epochs=600,
                    verbose=True,
                    shuffle=True,
                    callbacks=[lrScheduler],
                    validation_data=([X_test_0,X_test_1,X_test_2],Y_test)
                    )

# Plot training & validation accuracy values
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()


I am working on the file "DD_Net_demo_jhmdb_split2.ipynb".
The first task is:
Define configurations
Can you write Python code for it?

random.seed(1234)

class Config():
    def __init__(self):
        self.frame_l = 32 # the length of frames
        self.joint_n = 15 # the number of joints
        self.joint_d = 2 # the dimension of joints
        self.clc_num = 21 # the number of class
        self.feat_d = 105
        self.filters = 64
C = Config()


I am working on the file "DD_Net_demo_jhmdb_split2.ipynb".
The first task is:
Define data processing functions
Can you write Python code for it?


I am working on the file "DD_Net_demo_jhmdb_split2.ipynb".
The first task is:
Define network
Can you write Python code for it?


I am working on the file "DD_Net_demo_jhmdb_split2.ipynb".
The first task is:
Start train on split 1
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/BERD pretrained model/Named Entity Recognition With Bert.ipynb".
The first task is:
Before we can start the fine-tuning process, we have to setup the optimizer and add the parameters it should update. A common choice is the Adam optimizer. We also add some weight_decay as regularization to the main weight matrices. If you have limited resources, you can also try to just train the linear classifier on top of Bert and keep all other weights fixed. This will still give you a good performance.
Can you write Python code for it?

FULL_FINETUNING = True
if FULL_FINETUNING:
    param_optimizer = list(model.named_parameters())
    no_decay = ['bias', 'gamma', 'beta']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
         'weight_decay_rate': 0.01},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
         'weight_decay_rate': 0.0}
    ]
else:
    param_optimizer = list(model.classifier.named_parameters())
    optimizer_grouped_parameters = [{"params": [p for n, p in param_optimizer]}]
optimizer = Adam(optimizer_grouped_parameters, lr=3e-5)


I am working on the file "DEEP LEARNING/NLP/BERD pretrained model/Named Entity Recognition With Bert.ipynb".
The first task is:
Next, we cut and pad the token and label sequences to our desired length.
Can you write Python code for it?

input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts],
                          maxlen=MAX_LEN, dtype="long", truncating="post", padding="post")
tags = pad_sequences([[tag2idx.get(l) for l in lab] for lab in labels],
                     maxlen=MAX_LEN, value=tag2idx["O"], padding="post",
                     dtype="long", truncating="post")


I am working on the file "DEEP LEARNING/NLP/BERD pretrained model/Named Entity Recognition With Bert.ipynb".
The first task is:
Now we tokenize all sentences
Can you write Python code for it?

tokenized_texts = [tokenizer.tokenize(sent) for sent in sentences]
print(tokenized_texts[0])


I am working on the file "DEEP LEARNING/NLP/BERD pretrained model/Named Entity Recognition With Bert.ipynb".
The first task is:
We will limit our sequence length to 75 tokens and we will use a batch size of 32 as suggested by the Bert paper. Note, that Bert natively supports sequences of up to 512 tokens.
Can you write Python code for it?

MAX_LEN = 75
bs = 32

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
n_gpu = torch.cuda.device_count()

torch.cuda.get_device_name(0)


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
Hmm seems like numbers also are a problem. Lets check the top 10 embeddings to get a clue.
Can you write Python code for it?

for i in range(10):
    print(embeddings_index.index2entity[i])