SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Limitations of the proposed method. We implement the proposed inductive bias as a neural network layer that learns to scale and map points along a linear trajectory between (scaled) input features and their ranked counterparts. This implementation is motivated by our empirical analyses of commonly-used transformations in tabular deep learning, and is an attempt to generalize them while retaining their frequency-reducing properties. However, our implementation only covers a subset of such transformations. Indeed, one may carefully design transformations that guarantee frequency reduction with fewer negative side effects on the other training dynamics. Also, we limit
our experiments to independently transform each feature. More beneficial transformations might be found by instead first grouping features with informative interactions (i.e., subsets that are compatible to form informative higher-order features), then attempting to transform them to a low-frequency form. As tabular datasets feature heterogeneous feature spaces, identification of such compatible feature sets may be a challenge. It is possible that, for example, a pair of features may meaningfully interact in certain subregions (i.e., subdomains). Then, one may choose to identify these regions and represent them in different dimensions of the feature space. Such a mapping of features onto high-dimensional spaces to may help reduce target function frequency by transforming the feature space to be more homogeneous. We do not consider such approaches in this work. Finally, our methods only transform the inputs to the network, while more generally one may apply our frequency reduction inductive bias to all network activations to ensure that learned representations also have low-frequency form.
Limitations of the experiments. Our experiments mainly focus on observing the dataset-dependent effects of existing transformations that reduce target function frequency, and demonstrating how this dependency can be alleviated by using frequency reduction as an inductive bias during learning. In doing so, we limited our analysis to classification problems, although our methods can be applied in regression settings as well. Also, although our experimental setup enables a fair performance comparison between MLPs with different transformations, the corresponding results may not reflect the best performance achievable by MLPs. This is because we do not use techniques such as batch normalization, residual connections and weight decay regularizers that are shown to improve MLP performance on tabular data. This scope will be extended in future work. Our experiments with other neural network architectures are also limited to TabNet on binary classification datasets. Future work will also include extensive experiments with additional tabular data-specific deep learning architectures in order to explore the relationship between the improvements they offer and the frequency spectra of the functions they learn. Finally, although we qualitatively observe that our proposed methods alleviate the negative effects of spectral bias through our analysis of Fourier spectra and the corresponding impact on performance for all datasets considered, our quantification of spectral bias (see Appendix B.1) is limited to synthetic datasets due to its computational complexity. Such quantification can be extended beyond synthetic data experiments in the future.
H Implementation
In this section, we demonstrate how to implement the key components of our analyses and proposed method using PyTorch [20].
H.1 Proposed method as a network layer
The module CustomLayer implements our frequency-reducing transformation. In our implementation, in addition to scaling and combining, we also incorporate shifting factors in order to center features when necessary. We empirically observed that addition of the shifting factors did not change the performance of the neural network significantly. Notably, this implementation assumes that the input is in the form  $\mathbf{x} = [\mathbf{x}_{\mathrm{src}},\mathbf{x}_{\mathrm{trg}}] = [\mathbf{x}_{\mathrm{raw}},\mathbf{x}_{\mathrm{ranked}}]$ , where ranked is generated using scikit-learn's quantile transform. Also, for the "raw" features, our implementation uses their scaled version as an input to support training stability.
class CustomLayer(nnModule): # proposed layer
weights = torch.Tensor(torch.ones((1, size)) * 0.5)
scalingFactors_src = torch.Tensor(torch.ones((1, size)))
shifting Factors src = torch.Tensor(torch.zeros((1, size)))
scaling Factors trg = torch.Tensor(torch.ones((1, size)))
shifting Factors trg = torch.Tensor(torch.zeros((1, size)))
selfweights  $=$  nn.Parameters(weights)
selfscalingweights_src = nn_PARAMETER(scaling Factors_src)
selfshippingweights_src = nn.Parameters(shifting Factors_src)
selfscalingweights_trg = nn_PARAMETER(scaling Factors_trg)
self.shiftingweights_trg = nn.Parameters(shifting Factors_trg)
def forward(self, x):
    x_src = x[:, :x.size() [1] // 2] # raw input
    x_trg = x[:, x.size() [1] // 2:] # ranked input
    w = selfweights
    c_src = selfscalingweights_src
    u_src = self.shiftingweights_src
    c_trg = selfscalingweights_trg
    u_trg = self.shiftingweights_trg
    x = (((x_src + u_src) * c_src) * w) + (((x_trg + u_trg) * c_trg) * (1 - w))
    return x
class MLP(nnModule):
    def __init__(self, layers, activation_layer, p_dropout, bias, num_classes):
        layer_list = [nn.Linear(layers[i], layers[i+1], bias=bias) for i in range(len(layers)-1)]
        self.layers = nn.Module(layer_list)
        self.classifier = nn.Linear(layers[-1], num_classes)
        self_dropout = nn_dropout(p=p_dropout)
        self.activation_layer = activation_layer
        self(customLayer = CustomLayer(layers[0])
    def forward(self, x):
        x = self/customLayer(x)
        for i in range(len(self.layers)):
            x = self.layers[i](x)
            x = self.activation_layer(x)
            x = self_dropout(x)
        what = self.classifier(x)
    return what
H.2 Functions to calculate spectra
To efficiently compute NUDFT amplitudes, we utilize the nfft_adjoint function from the third party package nfft (https://github.com/jakevdp/nfft). The implementation is as follows:
!pip install nfft
from nfft import nfft_adjoint
def NUDFT(x, y, kmax, nk):
return (1/len(x)) * nfft_adjoint(-(x*kmax/nk), y, 2*(nk+1))[nk+1:]
def spectrum_NUDFT(x, y, kmax=50, nk=1000):
kvals = np.linspace(0, kmax, nk+1)
nufft = NUDFT(x, y, kmax, nk)
return [kvals, np.array(nufft, dtype="complex_")]
H.3 Selective-rank
In our experiments with selective rank, we set transformer to scikit-learn's quantile transform.
def get_features_with_reduced_freqency(X_tr, y_tr, transformer, threshold=0.75):
features = []
no_transform_features = []
X_tr_quantile = transformer.fit_transform(X_tr)  
for i in range(X_tr.shape[1]):  
    v_raw = spectrum(X_tr[:, i], y_tr) [lower_bound:]  
    v_qnt = spectrum(X_tr_quantile[:, i], y_tr) [lower_bound:]  
    metric = np.mean(abs(v_qnt) ** 2)/np.mean(abs(v_raw) ** 2)  
    if metric <= threshold:  
        features.append(i)  
    else:  
        no_transform_features.append(i)  
return features
def selective_transform(dataset,Scaler, config, transformer): X_tr, y_tr = dataset['tr'] ['X'], dataset['tr'] ['y'] X va, y va = dataset['va'] ['X'], dataset['va'] ['y'] X(te, y te = dataset['te'] ['X'], dataset['te'] ['y']) transform_cols = get_features_with_reduced_freqency(X_tr, y_tr, transformer, threshold=config['f_threshold']) ct = ColumnTransformer(['("sel-rank", transformer, transform_cols)]) ct.fit(X_tr, y_tr) X_tr:, transform_cols] = ct.transform(X_tr) X va:, transform_cols] = ct.transform(X va) X.te:, transform_cols] = ct.transform(X te) return X_tr, X(va, X(te
Figure 8: Weights learned by the proposed approach on all 14 datasets, part 1.
Figure 9: Weights learned by the proposed approach on all 14 datasets, part 2.