Title: No title found

Abstract: No abstract found

:

1 q(x) = (q 1 (x 1 ), . . . , q d (x d )),

q j m j q j,h (•) = 1 x j 2 C j,h , 0 1  h  m j , m j C j,h

x j x q j x j C j,h 1  h  m j C j,h = (c j,h 1 , c j,h ] c j,1 , . . . , c j,mj 1 c j,0 = 1 c j,mj = 1 m j = 3 c j,1 = 1/3 c j,2 = 2/3 q j (0.1) = (1, 0, 0)

x j x q j x j C j,h {1, . . . , l j } x j mj G h=1 C j,h = {1, . . . , l j }.

C j,1 = {1, 2} q j (1) = q j (2) = (1, 0, . . . , 0)

m j q j q |q| = d X j=1 m j . q Q Q d (m j ) d 1 l j 2 y 2 {0, 1} q(x) p ✓ (y|q(x)) |q| ✓ ✓ p ✓ (1|q(x)) 1 p ✓ (1|q(x)) ◆ = ✓ 0 + d X j=1 ✓ 0 j • q j (x j ), ✓ = (✓ 0 , (✓ j ) d 1 ) 2 R |q|+1 ✓ j = (✓ 1 j , . . . , ✓ mj j ) ✓ mj j = 0 j = 1 . . . d |q| m j ( , ) = (x 1 , . . . , x n ) = (y 1 , . . . , y n ) `q(✓; ( , )) = n X i=1 p ✓ (y i |q(x i )) ✓q ✓ q p ✓ q q = q2Q ⇢ `q( ✓q ; ( , )) 1 2 ⌫ q (n) ⌫ q ✓ q 2 Q d = 10 l j = 4 |Q| d 6 • 10 11 Q q j q q j q j,h

↵ j = (↵ j,1 , . . . , ↵ j,mj )

q ↵j (•) = q ↵ j,h (•) mj h=1 mj X h=1 q ↵ j,h (•) = 1 0  q ↵ j,h (•)  1, q ↵ j,h (•) h q ↵ j,h (•) ↵ = (↵ 1 , . . . , ↵ d ) q ↵ (•) = q ↵j (•) d j=1 Q q ↵ Q ⇢ e Q (q ↵, ✓) (q, ✓) q ↵ q ↵ j,h = (↵ 0 j,h , ↵ 1 j,h ) 2 R 2 q ↵ j,h (•) = (↵ 0 j,h + ↵ 1 j,h •) P mj g=1 (↵ 0 j,g + ↵ 1 j,g •) ↵ j,mj

(0, 0) ↵ j,h = (↵ j,h (1), . . . , ↵ j,h (l j )) 2 R lj q ↵ j,h (•) = (↵ j,h (•)) P mj g=1 (↵ j,g (•)) 000 [0.656, 0.666] [2.679, 2.941] [1.326, 1.554 ] n = 10, 000 [0.666, 0.666] [3.000, 3.000] [1.399, 1.621 We could do a similar graphic for categorical features.

l j x j ✓ p ✓ (1|q ↵ (x)) 1 p ✓ (1|q ↵ (x)) ◆ = ✓ 0 + d X j=1 ✓ 0 j • q ↵j (x j ), q q ↵ q ↵ q q ↵ q `q↵ (✓; ( , )) = n X i=1 p ✓ (y i |q ↵ (x i )) h = 1 . . . mj C j,h ↵ ✓ ( ↵, ✓) e Q Q q q ↵ qj,h (x j ) = 1 h = 1h 0 mj q ↵j,h 0 , 0 h x j C j,h ↵ ✓ p ✓ (1|q ↵ (x)) (•) x j x m j q ↵j (x j ) x j l j l j l j m j ⌃ ✓ x j 1 l j ⌃ ✓ ↵ j q ↵j (x j ) (↵, ✓) (t) q(t) ✓ (t) = ✓ `q(t) (✓; ( , )) (t) T t ⇤ = t2{1,...,T } (t) . m = (m j ) d 1 m q ↵ j,h (x i,j ) ⌧ 1 x i,j h qj,h = 0 m = m m m m x 1 x 2 [0, 1] q 1 (•) = q 2 (•) = (1 ] 1,1/3] (•), 1 ]1/3,2/3] (•), 1 ]2/3,1] (•)). d = 2 m 1 = m 2 = 3 c j,1 = 1/3 c j,2 = 2/3 j = 1, 2 ✓ = (0, 2, 2, 0, 2, 2, 0) y p ✓ (•|q(x)) ĉj,2 c j,2 = 2/3 m 1 = m 2 = 3 m1 m 1 = 3 m = 10 x 3 [0, 1] y m3 x 3 C j,h m j n = 48, 842 d = 14 ĉj,2 m1 m3 n = 1,

] ĉj,2 c j,2 = 2/3 m m 1 = 3 m3 m 3 = 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 q 0,1 q 0,2 q 0,3 x 1 q ↵ 0,h q ↵0,0 q ↵0,1 q ↵0,2 c 0,1 c 0,2 ĉ0,2 q(t) 1 (x1) t = 5 m = 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 q 0,1 q 0,2 q 0,3 x 1 q ↵ 0,h q ↵0,0 q ↵0,1 q ↵0,2 c 0,1 c 0,2 ĉ0,2 ĉ0,3 q(t) 1 (x1) t = 300 m = 3 q(t) 1 (x 1 ) n = 690 d = 14 n = 512 d = 39 n = 690 d = 15 n = 1, 000 d = 20 n = 270 d = 13 2 = 2 ⇥ 1 2 2 n =

4. Some Math:

We generated some data = (x i,j ) 1in,1jd1+d2 .

We preprocessed this data with a specified q ? = (q j ) d1 1 s.t.: 1  j  d1, q ? j (x j ) = (1 ] •;0.333] (x j ), 1 ]0.333;0.666] (x j ), 1 ]0.666,•[ (x j )) We then went on to prespecify a regression coefficient q ? and generated the target feature y s.t.:

Y ⇠ p q ? (•| f ? (x)) where ln ✓ p(1|q(x); q) 1 p(1|q(x); q) ◆ = q 0 + Â d1 j=1 q j 0 ⇥ q j (x j )

This gives us n responses = (y i ) n 1 . In practice, we are just given (, ) and wish to recover the "true" model q ?

, q ? . To do so in a finite sample setting, we give ourselves a model selection criterion like BIC, AIC, test set AUC, ... All discretization / grouping schemes q 2 Q cannot be tested (very high combinatorics) so that there is a need for additional assumptions / modelling + estimation procedure to reduce the computational burden and have at least some statistical guarantees (in n) to recover q ?

. N.B.: If the true data generating mechanism is not a discrete logistic regression, we get a representation of X that yields the "best" logistic regression (on discrete data w.r.t. a given criterion / loss), which is in practice very useful in the field of Credit Scoring (initial use case of this work).

We rely on a softmax approximation of q, i.e. q a j,h µ exp(a j,h,0 + a j,h,1 x j ), where the likelihood of (q, q) is "swapped" for (a, q). Refer to the paper for the proper justifications.

5. Neural network architecture:

To estimate q and a, we can use shallow neural networks by stating that: -If we allow q j to take m j values (we discretize in m j intervals), then we need m j softmax neurons per raw continuous feature x j . Other features shall not be connected to these neurons; -If we wish to group x j (categorical) into m j  o j levels, we one-hot encode each factor level and have an input neuron per level; we densely connect each of these neurons to m j softmax neurons.

Again, refer to the paper for the proper justifications.

In 

[i] = Input((1, )) liste_layers_quant[i] = Dense(m_quant[i], activation='softmax') liste_layers_quant_inputs[i] = liste_layers_quant[i](liste_inputs_quant[i])

In [19] : # A few utility functions are necessary to convert these parameters into # the pdf parametrized by alpha as described in the previous section.

# First, we transform the hidden layer's weights into a multinomial pdf K.function([liste_layers_quant[j] .input],

[liste_layers_quant [j] .output]) ( [x_quant[:, j, np. newaxis]])) if self.epochs in [1, 3, 5, 10, 20, 50, 100, 200] q 0,h q 0,1 q 0,2 q 0,3

Continuous feature 0 at iteration 1 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 1 1.0 q 0,h q 0,1 q 0,2 q 0,3

EQUATION

Continuous feature 0 at iteration 3 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 3 1.0 q 0,h q 0,1 q 0,2 q 0,3

EQUATION

Continuous feature 0 at iteration 5 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 5 1.0 q 0,h q 0,1 q 0,2 q 0,3

EQUATION

Continuous feature 0 at iteration 10 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 10 1.0 q 0,h q 0,1 q 0,2 q 0,3

EQUATION

Continuous feature 0 at iteration 20 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 20 q 1,0 q 1,1 q 1,2 q 1,3 q 1,4 c 1,1 c 1,2 ĉ1,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 0,h q 0,1 q 0,2 q 0,3

Continuous feature 0 at iteration 50 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 50 q 1,0 q 1,1 q 1,2 q 1,3 q 1,4 c 1,1 c 1,2 ĉ1,2 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 0,h q 0,1 q 0,2 q 0,3

Continuous feature 0 at iteration 100 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 ĉ0,3 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 100 q 1,0 q 1,1 q 1,2 q 1,3 q 1,4 c 1,1 c 1,2 ĉ1,2 ĉ1,3 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 0,h q 0,1 q 0,2 q 0,3

Continuous feature 0 at iteration 200 q 0,0 q 0,1 q 0,2 q 0,3 q 0,4 c 0,1 c 0,2 ĉ0,2 ĉ0,3 0.0 0.2 0.4 0.6 0.8 1.0

x 1 0.0 0.2 0.4 0.6 0.8 1.0 q 1,h q 1,1 q 1,2 q 1,3

Continuous feature 1 at iteration 200 q 1,0 q 1,1 q 1,2 q 1,3 q 1,4 c 1,1 c 1,2 ĉ1,2 ĉ1,3

Out[23]: <keras.callbacks.History at 0x1c2feaa390>



Figures and Tables:

]: # Second, we need to affect each discretizing / grouping to the argmax of # the previously calculated probabilities. def evaluate_disc(): proba = from_layers_to_proba_training() results = [None] * (d1) X_transformed = np.ones((n, 1))for j in range(d1): results[j] = np.argmax(proba[j][0], axis=1) X_transformed = np.concatenate( (X_transformed, sk.preprocessing.OneHotEncoder().fit_transform( X=results[j].reshape(-1, 1)).toarray()), axis=1) # Last we fit a logistic regression on the discretized / grouped features # and evaluate its BIC proposed_logistic_regression = sk.linear_model.LogisticRegression( fit_intercept=False) proposed_logistic_regression.fit(X=X_transformed, y=y.reshape((n, ))) BIC_training = 2 * sk.metrics.log_loss( y, proposed_logistic_regression.predict_proba(X=X_transformed)[:, 1], normalize=False ) + proposed_logistic_regression.coef_.shape[1] * np.log(n) return (BIC_training) In [21]: # We concatenate all the layers (each feature has its layer; layers are # in two lists depending on them being quantitative or qualitative). # We define the ouptput as a sigmoid (logistic regression). # We use an Adadelta optimizer. # We wish to optimize the log likelihood of the resulting model. from itertools import chain full_hidden = concatenate( list(chain.from_iterable([liste_layers_quant_inputs]))) output = Dense(1, activation='sigmoid')(full_hidden) model = Model( inputs=list(chain.from_iterable([liste_inputs_quant])), outputs=[output]) opt = optimizers.RMSprop(lr=0.5, rho=0.9, epsilon=None, decay=0.0) model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy']) In [22]: # At each training epoch, we must evaluate the quality of the derived class LossHistory(Callback): def on_train_begin(self, logs={}): self.losses = [] self.best_criterion = float("inf") self.best_outputs = [] self.epochs = 0 def on_epoch_end(self, batch, logs={}): self.losses.append(evaluate_disc()) self.epochs += 1 outputs = [] for j in range(d1): outputs.append(



Table content: <table><tr><td>n = 1000</td><td/></tr><tr><td>d1 = 2</td><td/></tr><tr><td>cuts = ([0, 0.333, 0.666, 1])</td><td/></tr><tr><td colspan="3">ICLR 2019 -Additional material to 'Feature quantization for In [4]: # x_quant is cut in thirds into xd_quant</td></tr><tr><td colspan="3">parsimonious and interpretable predictive models' x_quant = np.array(np.random.uniform(size=(n, d1)))</td></tr><tr><td>xd_quant = np.ndarray.copy(x_quant)</td><td/></tr><tr><td>for i in range(d1):</td><td/></tr><tr><td colspan="3">xd_quant[:, i] = pd.cut(x_quant[:, i], bins=cuts, labels=[0, 1, 2])</td></tr><tr><td colspan="2">November 23, 2018</td></tr><tr><td>2</td><td/></tr><tr><td>In [1]: import random</td><td/></tr><tr><td>import numpy as np</td><td/></tr><tr><td>import pandas as pd</td><td/></tr><tr><td>import sklearn as sk</td><td/></tr><tr><td>import sklearn.preprocessing</td><td/></tr><tr><td>import matplotlib.pyplot as plt</td><td/></tr><tr><td>plt.rc('text', usetex=True)</td><td/></tr><tr><td colspan="2">plt.rc('font', **{'family': "sans-serif"})</td></tr><tr><td>params = {</td><td/></tr><tr><td>'text.latex.preamble':</td><td/></tr><tr><td colspan="3">[r'\usepackage{amsmath}', r'\usepackage{amsfonts}', r'\usepackage{bm}']</td></tr><tr><td>}</td><td/></tr><tr><td>plt.rcParams.update(params)</td><td/></tr><tr><td colspan="2">from matplotlib2tikz import save as tikz_save</td><td>50, 000 n =</td></tr><tr><td>30, 000 n = 50, 000 n = 100, 000 n = 235, 000</td><td>n = 7, 500</td></tr><tr><td colspan="2">d = 25 d = 16 d = 15 d = 14 d = 14 In [2]: from IPython.display import set_matplotlib_formats d = 16 set_matplotlib_formats('pdf', 'png')</td><td>2 100</td></tr><tr><td>plt.rcParams['savefig.dpi'] = 75</td><td/></tr><tr><td>plt.rcParams['figure.autolayout'] = False</td><td/></tr><tr><td>plt.rcParams['figure.figsize'] = 10, 6</td><td/></tr><tr><td>plt.rcParams['axes.labelsize'] = 18</td><td/></tr><tr><td>plt.rcParams['axes.titlesize'] = 20</td><td/></tr><tr><td>plt.rcParams['font.size'] = 16</td><td/></tr><tr><td>plt.rcParams['lines.linewidth'] = 2.0</td><td/></tr><tr><td>plt.rcParams['lines.markersize'] = 8</td><td/></tr><tr><td>plt.rcParams['legend.fontsize'] = 14</td><td/></tr><tr><td>plt.rcParams['text.usetex'] = True</td><td/></tr><tr><td>plt.rcParams['font.family'] = "serif"</td><td/></tr><tr><td>plt.rcParams['font.serif'] = "cm"</td><td/></tr><tr><td colspan="2">In [3]: # n: sample size; d1: continuous features' dimension;</td></tr><tr><td colspan="2"># x_quant: U([0,1]); xd_quant: "true" generating data of Y</td></tr></table>



Table content: <table><tr><td/><td colspan="5">points_bokeh_y[:, 0], plt.hlines(0.666, 0, 1, colors='blue')</td><td/><td/><td/></tr><tr><td/><td colspan="5">marker='o', plt.vlines(0.333, 0, 1, colors='blue')</td><td/><td/><td/></tr><tr><td/><td colspan="5">s=points_bokeh_size[:, 0], plt.vlines(0.666, 0, 1, colors='blue')</td><td/><td/><td/></tr><tr><td/><td colspan="2">c="blue") plt.show()</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">plt.scatter(</td><td/><td/><td/><td/><td/><td/></tr><tr><td>1.0</td><td colspan="3">np.unique(log_odd), points_bokeh_y[:, 1],</td><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">marker='o',</td><td/><td/><td/><td/><td/><td/></tr><tr><td>0.8</td><td colspan="4">s=points_bokeh_size[:, 1], c="green")</td><td/><td/><td/><td/></tr><tr><td/><td colspan="7">plt.plot(np.unique(log_odd), np.unique(p), color="red")</td><td/></tr><tr><td>0.6</td><td>plt.show()</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>1.0</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.4</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.8</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.2</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.6</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.0</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.4</td><td>0.0</td><td>0.2</td><td/><td>0.4</td><td colspan="2">0.6</td><td>0.8</td><td/><td>1.0</td></tr><tr><td>0.2</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>0.0</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>4</td><td>3</td><td>2</td><td>1</td><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td></tr><tr><td colspan="10">In [10]: # The original value of the continuous features and their generated Y</td></tr><tr><td/><td colspan="9"># can be plotted; the provided cuts (.333 and .666) are clearly visible</td></tr><tr><td/><td colspan="7"># and separate different proportions of classes 0 and 1.</td><td/></tr><tr><td/><td colspan="9"># Clearly, such complicated boundaries could not be determined via</td></tr><tr><td/><td colspan="5"># a classical linear logistic regression.</td><td/><td/><td/></tr><tr><td/><td colspan="2">plt.scatter(</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="3">x_quant[0:10000, 0],</td><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="3">x_quant[0:10000, 1],</td><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">marker='o',</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="4">s=points_bokeh_size[:, 0],</td><td/><td/><td/><td/></tr><tr><td/><td colspan="7">c=(np.where(y[0:10000] == 0, "green", "red")))</td><td/></tr><tr><td/><td colspan="5">plt.hlines(0.333, 0, 1, colors='blue')</td><td/><td/><td/></tr></table>

[11]: # We rely on Keras for all calculations and the TensorFlow backend.

Table content: <table><tr><td>import tensorflow as tf</td></tr><tr><td>sess = tf.Session()</td></tr><tr><td>from keras import backend as K</td></tr><tr><td>K.set_session(sess)</td></tr><tr><td>/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the sec</td></tr><tr><td>from ._conv import register_converters as _register_converters</td></tr><tr><td>Using TensorFlow backend.</td></tr><tr><td>In [18]: # According to the previous section, each feature has its own</td></tr><tr><td># densely connected hidden layer</td></tr><tr><td>from keras import *</td></tr><tr><td>from keras.layers import *</td></tr><tr><td>from keras.callbacks import LambdaCallback, Callback, ReduceLROnPlateau, TensorBoard</td></tr><tr><td>import sklearn.linear_model</td></tr><tr><td>liste_inputs_quant = [None] * d1</td></tr><tr><td>liste_layers_quant = [None] * d1</td></tr><tr><td>liste_layers_quant_inputs = [None] * d1</td></tr><tr><td>m_quant = [5, 5]</td></tr><tr><td>for i in range(d1):</td></tr><tr><td>liste_inputs_quant</td></tr></table>

Peer review (iclr2019):
 The paper describes a question about discretize continuous features or group discrete features in the preprocessing step, which they call feature quantification. It considers that a joint training of feature quantification and a discriminative model can lead to a better performance than treating feature quantification as a preprocessing step. This paper has many typos, grammar mistakes and question marks, which make it hard to follow. The question proposed is simple and easy to understand. However, I don*t convinced by the solution in this paper. Since it is a hard optimization question, the authors proposed a relaxation approach in section 3.1. I do not think that exp(a+bx) is able to approach step functions since exp(a+bx) is monotone. I think Figure 1 is misleading. For grouping discrete features, the author propose to use exp(.alpha_{x_j, j}^h) and hoping that some .alpha parameters can be optimized to be equal, which is too simple. The exponential transformation here does not have an effect. It is more interesting to consider how to add some constraints. For example, if the discrete feature is ordinal, how one can assure that the grouped discrete feature is still ordinal. The relaxation in this paper is too much without handling any interesting constraints and the proposed exp(a+bx) can not approach step functions. The authors do not provide a good way to select number of cut points, which I think is a hard but interesting question. The work also lacks value in literature review, optimization and experiments.