Title: No title found

Abstract: No abstract found

:

1 q(x) = (q 1 (x 1 ), . . . , q d (x d )),

q j m j q j,h (•) = 1 x j 2 C j,h , 0 1  h  m j , m j C j,h

x j x q j x j C j,h 1  h  m j C j,h = (c j,h 1 , c j,h ] c j,1 , . . . , c j,mj 1 c j,0 = 1 c j,mj = 1 m j = 3 c j,1 = 1/3 c j,2 = 2/3 q j (0.1) = (1, 0, 0)

x j x q j x j C j,h {1, . . . , l j } x j mj G h=1 C j,h = {1, . . . , l j }.

C j,1 = {1, 2} q j (1) = q j (2) = (1, 0, . . . , 0)

m j q j q |q| = d X j=1 m j . q Q Q d (m j ) d 1 l j 2 y 2 {0, 1} q(x) p ✓ (y|q(x)) |q| ✓ ✓ p ✓ (1|q(x)) 1 p ✓ (1|q(x)) ◆ = ✓ 0 + d X j=1 ✓ 0 j • q j (x j ), ✓ = (✓ 0 , (✓ j ) d 1 ) 2 R |q|+1 ✓ j = (✓ 1 j , . . . , ✓ mj j ) ✓ mj j = 0 j = 1 . . . d |q| m j ( , ) = (x 1 , . . . , x n ) = (y 1 , . . . , y n ) `q(✓; ( , )) = n X i=1 p ✓ (y i |q(x i )) ✓q ✓ q p ✓ q q = q2Q ⇢ `q( ✓q ; ( , )) 1 2 ⌫ q (n) ⌫ q ✓ q 2 Q d = 10 l j = 4 |Q| d 6 • 10 11 Q q j q q j q j,h

↵ j = (↵ j,1 , . . . , ↵ j,mj )

q ↵j (•) = q ↵ j,h (•) mj h=1 mj X h=1 q ↵ j,h (•) = 1 0  q ↵ j,h (•)  1, q ↵ j,h (•) h q ↵ j,h (•) ↵ = (↵ 1 , . . . , ↵ d ) q ↵ (•) = q ↵j (•) d j=1 Q q ↵ Q ⇢ e Q (q ↵, ✓) (q, ✓) q ↵ q ↵ j,h = (↵ 0 j,h , ↵ 1 j,h ) 2 R 2 q ↵ j,h (•) = (↵ 0 j,h + ↵ 1 j,h •) P mj g=1 (↵ 0 j,g + ↵ 1 j,g •) ↵ j,mj

(0, 0) ↵ j,h = (↵ j,h (1), . . . , ↵ j,h (l j )) 2 R lj q ↵ j,h (•) = (↵ j,h (•)) P mj g=1 (↵ j,g (•)) 000 [0.656, 0.666] [2.679, 2.941] [1.326, 1.554 ] n = 10, 000 [0.666, 0.666] [3.000, 3.000] [1.399, 1.621 We could do a similar graphic for categorical features.

l j x j ✓ p ✓ (1|q ↵ (x)) 1 p ✓ (1|q ↵ (x)) ◆ = ✓ 0 + d X j=1 ✓ 0 j • q ↵j (x j ), q q ↵ q ↵ q q ↵ q `q↵ (✓; ( , )) = n X i=1 p ✓ (y i |q ↵ (x i )) h = 1 . . . mj C j,h ↵ ✓ ( ↵, ✓) e Q Q q q ↵ qj,h (x j ) = 1 h = 1h 0 mj q ↵j,h 0 , 0 h x j C j,h ↵ ✓ p ✓ (1|q ↵ (x)) (•) x j x m j q ↵j (x j ) x j l j l j l j m j ⌃ ✓ x j 1 l j ⌃ ✓ ↵ j q ↵j (x j ) (↵, ✓) (t) q(t) ✓ (t) = ✓ `q(t) (✓; ( , )) (t) T t ⇤ = t2{1,...,T } (t) . m = (m j ) d 1 m q ↵ j,h (x i,j ) ⌧ 1 x i,j h qj,h = 0 m = m m m m x 1 x 2 [0, 1] q 1 (•) = q 2 (•) = (1 ] 1,1/3] (•), 1 ]1/3,2/3] (•), 1 ]2/3,1] (•)). d = 2 m 1 = m 2 = 3 c j,1 = 1/3 c j,2 = 2/3 j = 1, 2 ✓ = (0, 2, 2, 0, 2, 2, 0) y p ✓ (•|q(x)) ĉj,2 c j,2 = 2/3 m 1 = m 2 = 3 m1 m 1 = 3 m = 10 x 3 [0, 1] y m3 x 3 C j,h m j n = 48, 842 d = 14 ĉj,2 m1 m3 n = 1,

] ĉj,2 c j,2 = 2/3 m m 1 = 3 m3 m 3 = 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 q 0,1 q 0,2 q 0,3 x 1 q ↵ 0,h q ↵0,0 q ↵0,1 q ↵0,2 c 0,1 c 0,2 ĉ0,2 q(t) 1 (x1) t = 5 m = 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 q 0,1 q 0,2 q 0,3 x 1 q ↵ 0,h q ↵0,0 q ↵0,1 q ↵0,2 c 0,1 c 0,2 ĉ0,2 ĉ0,3 q(t) 1 (x1) t = 300 m = 3 q(t) 1 (x 1 ) n = 690 d = 14 n = 512 d = 39 n = 690 d = 15 n = 1, 000 d = 20 n = 270 d = 13 2 = 2 ⇥ 1 2 2 n =

4. Some Math:

We generated some data = (x i,j ) 1in,1jd1+d2 .

We preprocessed this data with a specified q ? = (q j ) d1 1 s.t.: 1  j  d1, q ? j (x j ) = (1 ] •;0.333] (x j ), 1 ]0.333;0.666] (x j ), 1 ]0.666,•[ (x j )) We then went on to prespecify a regression coefficient q ? and generated the target feature y s.t.:

Y ⇠ p q ? (•| f ? (x)) where ln ✓ p(1|q(x); q) 1 p(1|q(x); q) ◆ = q 0 + Â d1 j=1 q j 0 ⇥ q j (x j )

This gives us n responses = (y i ) n 1 . In practice, we are just given (, ) and wish to recover the "true" model q ?

, q ? . To do so in a finite sample setting, we give ourselves a model selection criterion like BIC, AIC, test set AUC, ... All discretization / grouping schemes q 2 Q cannot be tested (very high combinatorics) so that there is a need for additional assumptions / modelling + estimation procedure to reduce the computational burden and have at least some statistical guarantees (in n) to recover q ?

. N.B.: If the true data generating mechanism is not a discrete logistic regression, we get a representation of X that yields the "best" logistic regression (on discrete data w.r.t. a given criterion / loss), which is in practice very useful in the field of Credit Scoring (initial use case of this work).

We rely on a softmax approximation of q, i.e. q a j,h µ exp(a j,h,0 + a j,h,1 x j ), where the likelihood of (q, q) is "swapped" for (a, q). Refer to the paper for the proper justifications.

5. Neural network architecture:

To estimate q and a, we can use shallow neural networks by stating that: -If we allow q j to take m j values (we discretize in m j intervals), then we need m j softmax neurons per raw continuous feature x j . Other features shall not be connected to these neurons; -If we wish to group x j (categorical) into m j  o j levels, we one-hot encode each factor level and have an input neuron per level; we densely connect each of these neurons to m j softmax neurons.

Again, refer to the paper for the proper justifications.

In 

[i] = Input((1, )) liste_layers_quant[i] = Dense(m_quant[i], activation='softmax') liste_layers_quant_inputs[i] = liste_layers_quant[i](liste_inputs_quant[i])

In [19] : # A few utility functions are necessary to convert these parameters into # the pdf parametrized by alpha as described in the previous section.

# First, we transform the hidden layer's weights into a multinomial pdf K.function([liste_layers_quant[j] .input],

[liste_layers_quant [j] .output]) ( [x_quant[:, j, np. newaxis]])) if self.epochs in [1, 3, 5, 10, 20, 50, 100, 200] q 0,h q 0,1 q 0,2 q 0,3

Continuous feature 0 at iteration 1 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 1 1.0 q 0,h q 0,1 q 0,2 q 0,3

EQUATION

Continuous feature 0 at iteration 3 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 3 1.0 q 0,h q 0,1 q 0,2 q 0,3

EQUATION

Continuous feature 0 at iteration 5 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 5 1.0 q 0,h q 0,1 q 0,2 q 0,3

EQUATION

Continuous feature 0 at iteration 10 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 10 1.0 q 0,h q 0,1 q 0,2 q 0,3

EQUATION

Continuous feature 0 at iteration 20 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 20 q 1,0 q 1,1 q 1,2 q 1,3 q 1,4 c 1,1 c 1,2 ĉ1,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 0,h q 0,1 q 0,2 q 0,3

Continuous feature 0 at iteration 50 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 50 q 1,0 q 1,1 q 1,2 q 1,3 q 1,4 c 1,1 c 1,2 ĉ1,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 0,h q 0,1 q 0,2 q 0,3

Continuous feature 0 at iteration 100 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 ĉ0,3 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 100 q 1,0 q 1,1 q 1,2 q 1,3 q 1,4 c 1,1 c 1,2 ĉ1,2 ĉ1,3 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 0,h q 0,1 q 0,2 q 0,3

Continuous feature 0 at iteration 200 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 ĉ0,3 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 200 q 1,0 q 1,1 q 1,2 q 1,3 q 1,4 c 1,1 c 1,2 ĉ1,2 ĉ1,3

Out[23]: <keras.callbacks.History at 0x1c2feaa390>



Figures and Tables:

]: # Second, we need to affect each discretizing / grouping to the argmax of # the previously calculated probabilities. def evaluate_disc(): proba = from_layers_to_proba_training() results = [None] * (d1) X_transformed = np.ones((n, 1))for j in range(d1): results[j] = np.argmax(proba[j][0], axis=1) X_transformed = np.concatenate( (X_transformed, sk.preprocessing.OneHotEncoder().fit_transform( X=results[j].reshape(-1, 1)).toarray()), axis=1) # Last we fit a logistic regression on the discretized / grouped features # and evaluate its BIC proposed_logistic_regression = sk.linear_model.LogisticRegression( fit_intercept=False) proposed_logistic_regression.fit(X=X_transformed, y=y.reshape((n, ))) BIC_training = 2 * sk.metrics.log_loss( y, proposed_logistic_regression.predict_proba(X=X_transformed)[:, 1], normalize=False ) + proposed_logistic_regression.coef_.shape[1] * np.log(n) return (BIC_training) In [21]: # We concatenate all the layers (each feature has its layer; layers are # in two lists depending on them being quantitative or qualitative). # We define the ouptput as a sigmoid (logistic regression). # We use an Adadelta optimizer. # We wish to optimize the log likelihood of the resulting model. from itertools import chain full_hidden = concatenate( list(chain.from_iterable([liste_layers_quant_inputs]))) output = Dense(1, activation='sigmoid')(full_hidden) model = Model( inputs=list(chain.from_iterable([liste_inputs_quant])), outputs=[output]) opt = optimizers.RMSprop(lr=0.5, rho=0.9, epsilon=None, decay=0.0) model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy']) In [22]: # At each training epoch, we must evaluate the quality of the derived class LossHistory(Callback): def on_train_begin(self, logs={}): self.losses = [] self.best_criterion = float("inf") self.best_outputs = [] self.epochs = 0 def on_epoch_end(self, batch, logs={}): self.losses.append(evaluate_disc()) self.epochs += 1 outputs = [] for j in range(d1): outputs.append(



Table content: <table><tr><td>n = 1000</td><td/></tr><tr><td>d1 = 2</td><td/></tr><tr><td>cuts = ([0, 0.333, 0.666, 1])</td><td/></tr><tr><td colspan="3">ICLR 2019 -Additional material to 'Feature quantization for In [4]: # x_quant is cut in thirds into xd_quant</td></tr><tr><td colspan="3">parsimonious and interpretable predictive models' x_quant = np.array(np.random.uniform(size=(n, d1)))</td></tr><tr><td>xd_quant = np.ndarray.copy(x_quant)</td><td/></tr><tr><td>for i in range(d1):</td><td/></tr><tr><td colspan="3">xd_quant[:, i] = pd.cut(x_quant[:, i], bins=cuts, labels=[0, 1, 2])</td></tr><tr><td colspan="2">November 23, 2018</td></tr><tr><td>2</td><td/></tr><tr><td>In [1]: import random</td><td/></tr><tr><td>import numpy as np</td><td/></tr><tr><td>import pandas as pd</td><td/></tr><tr><td>import sklearn as sk</td><td/></tr><tr><td>import sklearn.preprocessing</td><td/></tr><tr><td>import matplotlib.pyplot as plt</td><td/></tr><tr><td>plt.rc('text', usetex=True)</td><td/></tr><tr><td colspan="2">plt.rc('font', **{'family': "sans-serif"})</td></tr><tr><td>params = {</td><td/></tr><tr><td>'text.latex.preamble':</td><td/></tr><tr><td colspan="3">[r'\usepackage{amsmath}', r'\usepackage{amsfonts}', r'\usepackage{bm}']</td></tr><tr><td>}</td><td/></tr><tr><td>plt.rcParams.update(params)</td><td/></tr><tr><td colspan="2">from matplotlib2tikz import save as tikz_save</td><td>50, 000 n =</td></tr><tr><td>30, 000 n = 50, 000 n = 100, 000 n = 235, 000</td><td>n = 7, 500</td></tr><tr><td colspan="2">d = 25 d = 16 d = 15 d = 14 d = 14 In [2]: from IPython.display import set_matplotlib_formats d = 16 set_matplotlib_formats('pdf', 'png')</td><td>2 100</td></tr><tr><td>plt.rcParams['savefig.dpi'] = 75</td><td/></tr><tr><td>plt.rcParams['figure.autolayout'] = False</td><td/></tr><tr><td>plt.rcParams['figure.figsize'] = 10, 6</td><td/></tr><tr><td>plt.rcParams['axes.labelsize'] = 18</td><td/></tr><tr><td>plt.rcParams['axes.titlesize'] = 20</td><td/></tr><tr><td>plt.rcParams['font.size'] = 16</td><td/></tr><tr><td>plt.rcParams['lines.linewidth'] = 2.0</td><td/></tr><tr><td>plt.rcParams['lines.markersize'] = 8</td><td/></tr><tr><td>plt.rcParams['legend.fontsize'] = 14</td><td/></tr><tr><td>plt.rcParams['text.usetex'] = True</td><td/></tr><tr><td>plt.rcParams['font.family'] = "serif"</td><td/></tr><tr><td>plt.rcParams['font.serif'] = "cm"</td><td/></tr><tr><td colspan="2">In [3]: # n: sample size; d1: continuous features' dimension;</td></tr><tr><td colspan="2"># x_quant: U([0,1]); xd_quant: "true" generating data of Y</td></tr></table>



Table content: <table><tr><td/><td colspan="5">points_bokeh_y[:, 0], plt.hlines(0.666, 0, 1, colors='blue')</td><td/><td/><td/></tr><tr><td/><td colspan="5">marker='o', plt.vlines(0.333, 0, 1, colors='blue')</td><td/><td/><td/></tr><tr><td/><td colspan="5">s=points_bokeh_size[:, 0], plt.vlines(0.666, 0, 1, colors='blue')</td><td/><td/><td/></tr><tr><td/><td colspan="2">c="blue") plt.show()</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">plt.scatter(</td><td/><td/><td/><td/><td/><td/></tr><tr><td>1.0</td><td colspan="3">np.unique(log_odd), points_bokeh_y[:, 1],</td><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">marker='o',</td><td/><td/><td/><td/><td/><td/></tr><tr><td>0.8</td><td colspan="4">s=points_bokeh_size[:, 1], c="green")</td><td/><td/><td/><td/></tr><tr><td/><td colspan="7">plt.plot(np.unique(log_odd), np.unique(p), color="red")</td><td/></tr><tr><td>0.6</td><td>plt.show()</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>1.0</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.4</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.8</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.2</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.6</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.0</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.4</td><td>0.0</td><td>0.2</td><td/><td>0.4</td><td colspan="2">0.6</td><td>0.8</td><td/><td>1.0</td></tr><tr><td>0.2</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.0</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>4</td><td>3</td><td>2</td><td>1</td><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td></tr><tr><td colspan="10">In [10]: # The original value of the continuous features and their generated Y</td></tr><tr><td/><td colspan="9"># can be plotted; the provided cuts (.333 and .666) are clearly visible</td></tr><tr><td/><td colspan="7"># and separate different proportions of classes 0 and 1.</td><td/></tr><tr><td/><td colspan="9"># Clearly, such complicated boundaries could not be determined via</td></tr><tr><td/><td colspan="5"># a classical linear logistic regression.</td><td/><td/><td/></tr><tr><td/><td colspan="2">plt.scatter(</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="3">x_quant[0:10000, 0],</td><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="3">x_quant[0:10000, 1],</td><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">marker='o',</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="4">s=points_bokeh_size[:, 0],</td><td/><td/><td/><td/></tr><tr><td/><td colspan="7">c=(np.where(y[0:10000] == 0, "green", "red")))</td><td/></tr><tr><td/><td colspan="5">plt.hlines(0.333, 0, 1, colors='blue')</td><td/><td/><td/></tr></table>

[11]: # We rely on Keras for all calculations and the TensorFlow backend.

Table content: <table><tr><td>import tensorflow as tf</td></tr><tr><td>sess = tf.Session()</td></tr><tr><td>from keras import backend as K</td></tr><tr><td>K.set_session(sess)</td></tr><tr><td>/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the sec</td></tr><tr><td>from ._conv import register_converters as _register_converters</td></tr><tr><td>Using TensorFlow backend.</td></tr><tr><td>In [18]: # According to the previous section, each feature has its own</td></tr><tr><td># densely connected hidden layer</td></tr><tr><td>from keras import *</td></tr><tr><td>from keras.layers import *</td></tr><tr><td>from keras.callbacks import LambdaCallback, Callback, ReduceLROnPlateau, TensorBoard</td></tr><tr><td>import sklearn.linear_model</td></tr><tr><td>liste_inputs_quant = [None] * d1</td></tr><tr><td>liste_layers_quant = [None] * d1</td></tr><tr><td>liste_layers_quant_inputs = [None] * d1</td></tr><tr><td>m_quant = [5, 5]</td></tr><tr><td>for i in range(d1):</td></tr><tr><td>liste_inputs_quant</td></tr></table>

Peer review (iclr2019):
 This paper presents a feature quantization technique for logistic regression, which has already been a common practice in many finance applications. The text feels rushed. From the current presentation, I find it difficult to understand what is the motivation of adopting the proposed relaxation of the optimization method, and how is the neural network-based estimation strategy connected to the logistic regression model. It seems the difference lies in the parameterized nonlinear transformation such that the cutting points can be somehow optimized. The quality of the experiments performed is way below the expectation for ICLR. Although numerical experiments are performed on both simulated data and credit scoring data, it is still unclear whether the proposed method has superiority over competitors. Question: In the test phase, how would the proposed method handle features that are not seen in the training phase?