Repository: ludwig-ai/ludwig
Pull request: Encoder refactor V2
This PR contains all of Connor's commits from https://github.com/ludwig-ai/ludwig/pull/2269 (`:encoder_refactor`) merged atop the master branch, plus additional refactoring and test fixes in `:daniel/encoder_refactor`
```
master -----------------+------+>
                        |      |
daniel/encoder_refactor |      +*---->
                        |       | 
encoder_refactor        +------->

+ = branch
* = merge commit
```

A few notes:
- We have a glue function `load_config_with_kwargs` which loads a config from a dictionary, and collects any key-value pairs not captured by the config in an extra `kwargs` dictionary.  I removed this from instantiation of input features, now all input feature and encoder params are captured by the config.
- `load_config_with_kwargs` is still in use for combiners to pass in `output_size`.

Problem statement:
RFC: Nesting Refactor for Encoder/Decoder Config 
In the push to create a fully fledged schema using Marshmallow, we've run into a bit of an issue. When specifying things like optimizers, the structure of the config works well in a nested structure like this:
```
trainer:
     learning_rate: 0.001
     optimizer:
          type: adam
          beta_1: 0.9
          beta_2: 0.999
```
This works great for the structure of the schema since it allows us to create a hierarchical set of modules and submodules that can help build out the schema object which can be used internally for consistency in Ludwig, for populating parameter info on the frontend, and also for building the config object in the SDK. The issue here lies  in the config structure for Encoders and Decoders. Currently the way we specify encoder details in the config is without using any nesting and just adding any additional encoder specific parameters at the same level. For example:
```
input_features:
    - name: Feature_1
       type: binary
       encoder: dense
       dropout: 0.2
       activation: leakyRelu
    - name: Feature_2
       type: text
       encoder: stacked_cnn
       dropout: 0.2
       reduce_output: mean
       embeddings_on_cpu: false
       strides: 5
       output_size: 128
```
The issue here is that the encoder parameters are nested at the input feature level which is an issue for a few reasons:

- The code required to validate a schema without a nested module structure becomes very messy and convoluted which will only become more of a problem over time with the addition of new encoders and our value prop of extensibility.
- Specifying encoder parameters in the future SDK config object will be complicated and unintuitive because you will have to add the specific encoder parameters to the input feature as opposed the the specific encoder you want to use.

Because of this, we are proposing a refactor of the encoder config structure that nests encoders an additional level to stay consistent with a module structure. For example:

Instead of this 
```
input_features:
    - name: Feature_1
      type: binary
      encoder: dense
      dropout: 0.2
      activation: leakyRelu
```
We would do this
```
input_features:
    - name: Feature_1
      type: binary
      encoder: 
            type: dense
            dropout: 0.2
            activation: leakyRelu
```

Note this also applies for decoders as well. If we decide to move forward on this proposal, we would need to put in some logic to handle backwards compatibility for all the existing Ludwig users. While this may seem like a headache, I believe that this is something that we definitely want to do, and I think that doing it now as opposed to later is a good idea since the impact issues from a change like this will only increase over time. Let me know your thoughts!

Hints from issue discussion:
I'm not necessarily opposed to this change, but this would be a rather significant backwards compatibility layer that we'd need to maintain for a long time, so definitely want to verify that this is absolutely something we want to do. 

> * Specifying encoder parameters in the future SDK config object will be complicated and unintuitive because you will have to add the specific encoder parameters to the input feature as opposed the the specific encoder you want to use.

We had a [discussion](https://ludwig-ai.slack.com/archives/C02CK64TKGQ/p1643682586977809) about whether to flatten or nest for input features a while back. Here's @w4nderlust's comment:

> **Not nested encoder/decoder parameters:** there’s merit to both indentation and not indentation. At the moment I’d like to keep it not indented. The main motivation being that there are some parameters in the config passed to the encoders that are added depending on data preprocessing (for instance the vocab size of a categorical embed encoder) and those may be needed by both encoders and decoders and potentially other functions within the input/output feature object and after trying for a few minutes to implement this there doesn’t seem a simple way not to break encapsulation without a substantial rework. None of this would be unsolvable, but would require more reworking than it seems at first glance, so I don’t think this is the right time for this additional reworking, so we should reconsider later on if we want to revisit this. From a PQL perspective, I think it’s fine to have encoder and decoder parameters on their own and then flatten them before passing to Ludwig. I believe the nesting is a better idea in general, don’t get me wrong, but would want to explore thoroughly nuances and corner case in the code and its impact on the config verbosity more in depth.

I think the idea is that the feature module is the sole owner of its encoder submodule. The fact that the same parameters are shared between a feature module that instantiates a submodule with those parameters is an implementation detail.

As Piero mentioned, there could be some additional (internal-only) parameters that are decided at preprocessing and then passed to the encoder/decoder submodule, and this is managed at the feature level -- but this is also an implementation detail.

By definition, then, there aren't any parameters (except for preprocessing which has its own subsection) that would be used exclusively at the feature level and not at the encoder submodule level. 

> * The code required to validate a schema without a nested module structure becomes very messy and convoluted which will only become more of a problem over time with the addition of new encoders and our value prop of extensibility.

@ksbrar How difficult is it to validate a schema without the nested parameter structure? I imagine that we only need what we're already doing for the `combiner`, `trainer, and `optimizer` sections, which is to use the special `type` parameter to dictate how to validate the rest of the declared parameters within the section, i.e.

```yaml
combiner:
  type: concat
  <list of parameters that should be validated against the `ConcatCombiner` combiner schema>

optimizer:
  type: adamw
  <list of parameters that should be validated against the `adamw` optimizer schema>

trainer:
  type: GBM
  <list of parameters that should be validated against the `GBM` trainer schema>
```

Possibilities that would convince me that nesting is absolutely necessary:
- There actually are parameters (except for preprocessing/internal) that would be used at the feature level and not the encoder submodule.
- Validation is significantly less difficult if we do parameter nesting.
- The nesting has structural value that has strong external value beyond what's reasonable to manage internally.

Open to more thoughts on this. @dantreiman @w4nderlust @tgaddair @brightsparc 
@ksbrar chime in here if I'm missing something, but my understanding is that schema validation without the nested parameter structure would just require a lot of marshmallow `validates_schema()` logic. Since we essentially would house all the possible encoder parameters from all encoders available to a certain input feature type in that same input feature schema, we would need a complex validation function that essentially makes sure that you are not only specifying the encoder params specific to a particular encoder type. While this is entirely possible, I foresee this getting pretty complicated in the future if we end up adding any additional encoders to certain feature types.

The bigger reason I see to upgrade personally is how this schema object that we're building out maps over to the SDK. Since we want to have a config object that the user interacts with the build their models, the way that our schema is structured is essentially what the customer will be touching when using the SDK. So not only do we want this to be consistent across modules, but also we want it to be intuitive to use. If we choose to not refactor the schema and nest the encoder/decoder structure, there are potential confusion issues with customers trying to specify encoders as part of their input feature and not knowing which items to include and which not to. I also know that something we support is extensibility for encoders and while this is an area where I'm not very familiar with, I feel like a nested structure would lend well to incorporating user defined encoders. For example:
```
input_features:
    - name: feature_1
       type: categorical
       encoder:
              type: user_defined_encoder
              param_1: ...
              param_2: ...
```
Something like this makes a lot of sense from a user perspective IMO. Additionally, I imagine down the road we could even be able to create adaptable logic that can validate the schema for a user defined encoder. 

Anyways, I greatly appreciate your feedback and would love to hear others thoughts as well so we can decide whether we want to go through with this or not:)
Different proposals:

```
# 1. This
LudwigConfig(input_features=[
   BinaryInputFeature(name="foo", encoder=Dense(num_layers=3), preprocessing=...)
])

# 2. No config changes option A
LudwigConfig(input_features=[
   BinaryInputDenseEncoder(name="foo", num_layers=3, preprocessing=...)
])

# 3. No config changes option B
LudwigConfig(input_features=[
    BinaryInputFeatuere(name="foo", encoder="dense", num_layers=3)
])
```
Current thoughts:

- Generally like the nested structure
- Option 2 may be the best compromise for now
- Worth investigation to see how hard option 1 would be, as it is the longterm solution
Maybe also be worth thinking about how this overlays with current PQL which is inline with option 1. Also UX more closely matches this IMHO. My understanding on original push back when I raised this idea was the challenge of changing the Ludwig codebase to support this - could we perhaps confirm this again? 
@brightsparc we discussed the challenge of implementing the nested structure in terms of the refactoring effort. The consensus is that we should explore it and see where the complexity exists, if it does, and how much effort would be required to resolve it.
I plan on scoping out the difficulty of a refactor after I have worked through option 2 to fill out the schema in the mean time.

Commit message: [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Patch to review:
diff --git a/ludwig/combiners/combiners.py b/ludwig/combiners/combiners.py
index 666afddd0bc..3d833c31074 100644
--- a/ludwig/combiners/combiners.py
+++ b/ludwig/combiners/combiners.py
@@ -29,16 +29,14 @@
 from ludwig.modules.fully_connected_modules import FCStack
 from ludwig.modules.reduction_modules import SequenceReducer
 from ludwig.modules.tabnet_modules import TabNet
-from ludwig.schema.combiners import (
-    ComparatorCombinerConfig,
-    ConcatCombinerConfig,
-    ProjectAggregateCombinerConfig,
-    SequenceCombinerConfig,
-    SequenceConcatCombinerConfig,
-    TabNetCombinerConfig,
-    TabTransformerCombinerConfig,
-    TransformerCombinerConfig,
-)
+from ludwig.schema.combiners.comparator import ComparatorCombinerConfig
+from ludwig.schema.combiners.concat import ConcatCombinerConfig
+from ludwig.schema.combiners.project_aggregate import ProjectAggregateCombinerConfig
+from ludwig.schema.combiners.sequence import SequenceCombinerConfig
+from ludwig.schema.combiners.sequence_concat import SequenceConcatCombinerConfig
+from ludwig.schema.combiners.tab_transformer import TabTransformerCombinerConfig
+from ludwig.schema.combiners.tabnet import TabNetCombinerConfig
+from ludwig.schema.combiners.transformer import TransformerCombinerConfig
 from ludwig.schema.combiners.utils import combiner_registry, get_combiner_jsonschema, register_combiner
 from ludwig.utils.misc_utils import get_from_registry
 from ludwig.utils.torch_utils import LudwigModule, sequence_length_3D
diff --git a/ludwig/constants.py b/ludwig/constants.py
index 8a90887859b..3da862676d9 100644
--- a/ludwig/constants.py
+++ b/ludwig/constants.py
@@ -13,6 +13,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
+INPUT_FEATURES = "input_features"
+OUTPUT_FEATURES = "output_features"
+
 BINARY = "binary"
 CATEGORY = "category"
 INT = "int"
@@ -40,9 +43,11 @@
 ROC_AUC = "roc_auc"
 EVAL_LOSS = "eval_loss"
 TRAIN_MEAN_LOSS = "train_mean_loss"
+SEQUENCE_SOFTMAX_CROSS_ENTROPY = "sequence_softmax_cross_entropy"
 SOFTMAX_CROSS_ENTROPY = "softmax_cross_entropy"
 SIGMOID_CROSS_ENTROPY = "sigmoid_cross_entropy"
 BINARY_WEIGHTED_CROSS_ENTROPY = "binary_weighted_cross_entropy"
+THRESHOLD = "threshold"
 ACCURACY = "accuracy"
 HITS_AT_K = "hits_at_k"
 MEAN_HITS_AT_K = "mean_hits_at_k"
@@ -58,6 +63,7 @@
 PERPLEXITY = "perplexity"
 JACCARD = "jaccard"
 PREDICTIONS = "predictions"
+TOP_K = "top_k"
 TOP_K_PREDICTIONS = "top_k_predictions"
 PROBABILITY = "probability"
 PROBABILITIES = "probabilities"
@@ -136,9 +142,6 @@
 SCHEDULER = "scheduler"
 PARAMETERS = "parameters"
 
-INPUT_FEATURES = "input_features"
-OUTPUT_FEATURES = "output_features"
-
 NAME = "name"
 COLUMN = "column"
 TYPE = "type"
@@ -167,12 +170,18 @@
 AUTO = "auto"
 CONFIG = "config"
 
+CLIP = "clip"
+DEPENDENCIES = "dependencies"
+REDUCE_INPUT = "reduce_input"
+REDUCE_DEPENDENCIES = "reduce_dependencies"
+
 COMBINER = "combiner"
 
 ENCODER = "encoder"
 DECODER = "decoder"
 
 DEFAULTS = "defaults"
+DEFAULT = "default"
 
 BALANCE_PERCENTAGE_TOLERANCE = 0.03
 IMBALANCE_DETECTION_RATIO = 0.05
diff --git a/ludwig/data/dataset_synthesizer.py b/ludwig/data/dataset_synthesizer.py
index 855df25e560..5ab4822d7a6 100644
--- a/ludwig/data/dataset_synthesizer.py
+++ b/ludwig/data/dataset_synthesizer.py
@@ -33,6 +33,8 @@
     BINARY,
     CATEGORY,
     DATE,
+    DECODER,
+    ENCODER,
     H3,
     IMAGE,
     NAME,
@@ -111,7 +113,12 @@ def return_none(feature):
 
 
 def assign_vocab(feature):
-    feature["idx2str"] = build_vocab(feature.get("vocab_size", 10))
+    if DECODER in feature:
+        feature["idx2str"] = build_vocab(feature[DECODER].get("vocab_size", 10))
+    elif ENCODER in feature:
+        feature["idx2str"] = build_vocab(feature[ENCODER].get("vocab_size", 10))
+    else:
+        feature["idx2str"] = build_vocab(10)
 
 
 def build_feature_parameters(features):
@@ -371,7 +378,11 @@ def generate_h3(feature):
 
 def generate_vector(feature):
     # Space delimited string with floating point numbers
-    return " ".join([str(100 * random.random()) for _ in range(feature.get("vector_size", 10))])
+    if PREPROCESSING in feature:
+        vector_size = feature[PREPROCESSING].get("vector_size", 10)
+    else:
+        vector_size = feature.get("vector_size", 10)
+    return " ".join([str(100 * random.random()) for _ in range(vector_size)])
 
 
 generators_registry = {
diff --git a/ludwig/data/preprocessing.py b/ludwig/data/preprocessing.py
index 2e24f764aeb..640edd2e51f 100644
--- a/ludwig/data/preprocessing.py
+++ b/ludwig/data/preprocessing.py
@@ -1265,18 +1265,19 @@ def build_preprocessing_parameters(
 
         # deal with encoders that have fixed preprocessing
         if ENCODER in feature_config:
-            encoder_class = get_encoder_cls(feature_config[TYPE], feature_config["encoder"])
-            if hasattr(encoder_class, "fixed_preprocessing_parameters"):
-                encoder_fpp = encoder_class.fixed_preprocessing_parameters
+            if TYPE in feature_config[ENCODER]:
+                encoder_class = get_encoder_cls(feature_config[TYPE], feature_config[ENCODER][TYPE])
+                if hasattr(encoder_class, "fixed_preprocessing_parameters"):
+                    encoder_fpp = encoder_class.fixed_preprocessing_parameters
 
-                preprocessing_parameters = merge_dict(
-                    preprocessing_parameters, resolve_pointers(encoder_fpp, feature_config, "feature.")
-                )
+                    preprocessing_parameters = merge_dict(
+                        preprocessing_parameters, resolve_pointers(encoder_fpp, feature_config, "feature.")
+                    )
 
         fill_value = precompute_fill_value(dataset_cols, feature_config, preprocessing_parameters, backend)
 
         if fill_value is not None:
-            preprocessing_parameters = {"computed_fill_value": fill_value, **preprocessing_parameters}
+            preprocessing_parameters.update({"computed_fill_value": fill_value})
 
         feature_name_to_preprocessing_parameters[feature_name] = preprocessing_parameters
 
@@ -1443,11 +1444,14 @@ def handle_missing_values(dataset_cols, feature, preprocessing_parameters):
     # Check for the precomputed fill value in the metadata
     computed_fill_value = preprocessing_parameters.get("computed_fill_value")
 
-    if computed_fill_value is not None:
+    if (
+        missing_value_strategy in {FILL_WITH_CONST, FILL_WITH_MODE, FILL_WITH_MEAN, FILL_WITH_FALSE}
+        and computed_fill_value is not None
+    ):
         dataset_cols[feature[COLUMN]] = dataset_cols[feature[COLUMN]].fillna(
             computed_fill_value,
         )
-    elif missing_value_strategy in [BACKFILL, BFILL, PAD, FFILL]:
+    elif missing_value_strategy in {BACKFILL, BFILL, PAD, FFILL}:
         dataset_cols[feature[COLUMN]] = dataset_cols[feature[COLUMN]].fillna(
             method=missing_value_strategy,
         )
@@ -1457,7 +1461,7 @@ def handle_missing_values(dataset_cols, feature, preprocessing_parameters):
         # result in the removal of the rows.
         dataset_cols[feature[COLUMN]] = dataset_cols[feature[COLUMN]].dropna()
     else:
-        raise ValueError("Invalid missing value strategy")
+        raise ValueError(f"Invalid missing value strategy {missing_value_strategy}")
 
 
 def load_hdf5(hdf5_file_path, preprocessing_params, backend, split_data=True, shuffle_training=False):
diff --git a/ludwig/decoders/generic_decoders.py b/ludwig/decoders/generic_decoders.py
index 9fdda4b47a0..11b249a7f44 100644
--- a/ludwig/decoders/generic_decoders.py
+++ b/ludwig/decoders/generic_decoders.py
@@ -21,6 +21,7 @@
 from ludwig.constants import BINARY, CATEGORY, LOSS, NUMBER, SEQUENCE, SET, TEXT, TYPE, VECTOR
 from ludwig.decoders.base import Decoder
 from ludwig.decoders.registry import register_decoder
+from ludwig.schema.decoders.base import ClassifierConfig, PassthroughDecoderConfig, ProjectorConfig, RegressorConfig
 from ludwig.utils.torch_utils import Dense, get_activation
 
 logger = logging.getLogger(__name__)
@@ -37,6 +38,10 @@ def __init__(self, input_size: int = 1, num_classes: int = None, **kwargs):
     def forward(self, inputs, **kwargs):
         return inputs
 
+    @staticmethod
+    def get_schema_cls():
+        return PassthroughDecoderConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.input_size])
@@ -46,7 +51,7 @@ def output_shape(self) -> torch.Size:
         return self.input_shape
 
 
-@register_decoder("regressor", [BINARY, NUMBER], default=True)
+@register_decoder("regressor", [BINARY, NUMBER])
 class Regressor(Decoder):
     def __init__(
         self,
@@ -70,6 +75,10 @@ def __init__(
             bias_initializer=bias_initializer,
         )
 
+    @staticmethod
+    def get_schema_cls():
+        return RegressorConfig
+
     @property
     def input_shape(self):
         return self.dense.input_shape
@@ -78,7 +87,7 @@ def forward(self, inputs, **kwargs):
         return self.dense(inputs)
 
 
-@register_decoder("projector", [VECTOR], default=True)
+@register_decoder("projector", [VECTOR])
 class Projector(Decoder):
     def __init__(
         self,
@@ -116,6 +125,10 @@ def __init__(
         else:
             self.clip = None
 
+    @staticmethod
+    def get_schema_cls():
+        return ProjectorConfig
+
     @property
     def input_shape(self):
         return self.dense.input_shape
@@ -127,7 +140,7 @@ def forward(self, inputs, **kwargs):
         return values
 
 
-@register_decoder("classifier", [CATEGORY, SET], default=True)
+@register_decoder("classifier", [CATEGORY, SET])
 class Classifier(Decoder):
     def __init__(
         self,
@@ -160,6 +173,10 @@ def __init__(
         # otherwise the weights of the Dense layer would not be initialized
         self.first_call = True
 
+    @staticmethod
+    def get_schema_cls():
+        return ClassifierConfig
+
     @property
     def input_shape(self):
         return self.dense.input_shape
diff --git a/ludwig/decoders/registry.py b/ludwig/decoders/registry.py
index 640b5ec3d20..e57bf50de91 100644
--- a/ludwig/decoders/registry.py
+++ b/ludwig/decoders/registry.py
@@ -1,12 +1,12 @@
 from typing import Dict, List, Type, Union
 
 from ludwig.decoders.base import Decoder
-from ludwig.utils.registry import DEFAULT_KEYS, Registry
+from ludwig.utils.registry import Registry
 
 decoder_registry = Registry()
 
 
-def register_decoder(name: str, features: Union[str, List[str]], default=False):
+def register_decoder(name: str, features: Union[str, List[str]]):
     if isinstance(features, str):
         features = [features]
 
@@ -14,9 +14,6 @@ def wrap(cls):
         for feature in features:
             feature_registry = decoder_registry.get(feature, {})
             feature_registry[name] = cls
-            if default:
-                for key in DEFAULT_KEYS:
-                    feature_registry[key] = cls
             decoder_registry[feature] = feature_registry
         return cls
 
diff --git a/ludwig/decoders/sequence_decoders.py b/ludwig/decoders/sequence_decoders.py
index 762c2a2f5c0..809ac55ee90 100644
--- a/ludwig/decoders/sequence_decoders.py
+++ b/ludwig/decoders/sequence_decoders.py
@@ -23,6 +23,7 @@
 from ludwig.decoders.registry import register_decoder
 from ludwig.decoders.sequence_decoder_utils import get_lstm_init_state, get_rnn_init_state
 from ludwig.modules.reduction_modules import SequenceReducer
+from ludwig.schema.decoders.sequence_decoders import SequenceGeneratorDecoderConfig
 from ludwig.utils import strings_utils
 
 logger = logging.getLogger(__name__)
@@ -313,6 +314,10 @@ def forward(
     def get_prediction_set(self):
         return {LOGITS}
 
+    @staticmethod
+    def get_schema_cls():
+        return SequenceGeneratorDecoderConfig
+
     @property
     def input_shape(self):
         # Dummy implementation.
diff --git a/ludwig/decoders/sequence_tagger.py b/ludwig/decoders/sequence_tagger.py
index d321a3475be..2b78e0dda09 100644
--- a/ludwig/decoders/sequence_tagger.py
+++ b/ludwig/decoders/sequence_tagger.py
@@ -7,6 +7,7 @@
 from ludwig.decoders.base import Decoder
 from ludwig.decoders.registry import register_decoder
 from ludwig.modules.attention_modules import MultiHeadSelfAttention
+from ludwig.schema.decoders.sequence_decoders import SequenceTaggerDecoderConfig
 from ludwig.utils.torch_utils import Dense
 
 logger = logging.getLogger(__name__)
@@ -73,6 +74,10 @@ def forward(self, inputs: Dict[str, torch.Tensor], target: torch.Tensor = None)
     def get_prediction_set(self):
         return {LOGITS}
 
+    @staticmethod
+    def get_schema_cls():
+        return SequenceTaggerDecoderConfig
+
     @property
     def input_shape(self):
         # Dummy implementation.
diff --git a/ludwig/encoders/bag_encoders.py b/ludwig/encoders/bag_encoders.py
index e51c0161810..cbfcc640c81 100644
--- a/ludwig/encoders/bag_encoders.py
+++ b/ludwig/encoders/bag_encoders.py
@@ -23,11 +23,12 @@
 from ludwig.encoders.registry import register_encoder
 from ludwig.modules.embedding_modules import EmbedWeighted
 from ludwig.modules.fully_connected_modules import FCStack
+from ludwig.schema.encoders.bag_encoders import BagEmbedWeightedConfig
 
 logger = logging.getLogger(__name__)
 
 
-@register_encoder("embed", BAG, default=True)
+@register_encoder("embed", BAG)
 class BagEmbedWeightedEncoder(Encoder):
     def __init__(
         self,
@@ -80,6 +81,10 @@ def __init__(
             default_dropout=dropout,
         )
 
+    @staticmethod
+    def get_schema_cls():
+        return BagEmbedWeightedConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([len(self.vocab)])
diff --git a/ludwig/encoders/binary_encoders.py b/ludwig/encoders/binary_encoders.py
index f31ded09db0..d0efd00bcb5 100644
--- a/ludwig/encoders/binary_encoders.py
+++ b/ludwig/encoders/binary_encoders.py
@@ -20,11 +20,12 @@
 from ludwig.constants import BINARY
 from ludwig.encoders.base import Encoder
 from ludwig.encoders.registry import register_encoder
+from ludwig.schema.encoders.binary_encoders import BinaryPassthroughEncoderConfig
 
 logger = logging.getLogger(__name__)
 
 
-@register_encoder("passthrough", BINARY, default=True)
+@register_encoder("passthrough", BINARY)
 class BinaryPassthroughEncoder(Encoder):
     def __init__(self, **kwargs):
         super().__init__()
@@ -40,6 +41,10 @@ def forward(self, inputs: torch.Tensor) -> torch.Tensor:
 
         return inputs
 
+    @staticmethod
+    def get_schema_cls():
+        return BinaryPassthroughEncoderConfig
+
     @property
     def output_shape(self) -> torch.Size:
         return torch.Size([1])
diff --git a/ludwig/encoders/category_encoders.py b/ludwig/encoders/category_encoders.py
index e8cc4893dd6..7433620fd98 100644
--- a/ludwig/encoders/category_encoders.py
+++ b/ludwig/encoders/category_encoders.py
@@ -22,6 +22,7 @@
 from ludwig.encoders.base import Encoder
 from ludwig.encoders.registry import register_encoder
 from ludwig.modules.embedding_modules import Embed
+from ludwig.schema.encoders.category_encoders import CategoricalEmbedConfig, CategoricalSparseConfig
 
 logger = logging.getLogger(__name__)
 
@@ -65,6 +66,10 @@ def forward(self, inputs: torch.Tensor) -> torch.Tensor:
         embedded = self.embed(inputs)
         return embedded
 
+    @staticmethod
+    def get_schema_cls():
+        return CategoricalEmbedConfig
+
     @property
     def output_shape(self) -> torch.Size:
         return torch.Size([self.embedding_size])
@@ -112,6 +117,10 @@ def forward(self, inputs: torch.Tensor) -> torch.Tensor:
         embedded = self.embed(inputs)
         return embedded
 
+    @staticmethod
+    def get_schema_cls():
+        return CategoricalSparseConfig
+
     @property
     def output_shape(self) -> torch.Size:
         return torch.Size([self.embedding_size])
diff --git a/ludwig/encoders/date_encoders.py b/ludwig/encoders/date_encoders.py
index 0583feaa894..1e0ec0b18fb 100644
--- a/ludwig/encoders/date_encoders.py
+++ b/ludwig/encoders/date_encoders.py
@@ -23,6 +23,7 @@
 from ludwig.encoders.registry import register_encoder
 from ludwig.modules.embedding_modules import Embed
 from ludwig.modules.fully_connected_modules import FCStack
+from ludwig.schema.encoders.date_encoders import DateEmbedConfig, DateWaveConfig
 from ludwig.utils import torch_utils
 
 logger = logging.getLogger(__name__)
@@ -258,6 +259,10 @@ def forward(self, inputs: torch.Tensor) -> Dict[str, torch.Tensor]:
         hidden = self.fc_stack(hidden)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return DateEmbedConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([DATE_INPUT_SIZE])
@@ -383,6 +388,10 @@ def forward(self, inputs: torch.Tensor) -> Dict[str, torch.Tensor]:
         hidden = self.fc_stack(hidden)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return DateWaveConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([DATE_INPUT_SIZE])
diff --git a/ludwig/encoders/generic_encoders.py b/ludwig/encoders/generic_encoders.py
index 4a909227879..3247ef3c673 100644
--- a/ludwig/encoders/generic_encoders.py
+++ b/ludwig/encoders/generic_encoders.py
@@ -21,11 +21,12 @@
 from ludwig.encoders.base import Encoder
 from ludwig.encoders.registry import register_encoder
 from ludwig.modules.fully_connected_modules import FCStack
+from ludwig.schema.encoders.base import DenseEncoderConfig, PassthroughEncoderConfig
 
 logger = logging.getLogger(__name__)
 
 
-@register_encoder("passthrough", [CATEGORY, NUMBER, VECTOR], default=True)
+@register_encoder("passthrough", [CATEGORY, NUMBER, VECTOR])
 class PassthroughEncoder(Encoder):
     def __init__(self, input_size=1, **kwargs):
         super().__init__()
@@ -39,6 +40,10 @@ def forward(self, inputs, mask=None):
         """
         return {"encoder_output": inputs}
 
+    @staticmethod
+    def get_schema_cls():
+        return PassthroughEncoderConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.input_size])
@@ -91,6 +96,10 @@ def forward(self, inputs, training=None, mask=None):
         """
         return {"encoder_output": self.fc_stack(inputs)}
 
+    @staticmethod
+    def get_schema_cls():
+        return DenseEncoderConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.input_size])
diff --git a/ludwig/encoders/h3_encoders.py b/ludwig/encoders/h3_encoders.py
index 851bb7e4ab0..a8ea77645da 100644
--- a/ludwig/encoders/h3_encoders.py
+++ b/ludwig/encoders/h3_encoders.py
@@ -26,6 +26,7 @@
 from ludwig.modules.initializer_modules import get_initializer
 from ludwig.modules.recurrent_modules import RecurrentStack
 from ludwig.modules.reduction_modules import SequenceReducer
+from ludwig.schema.encoders.h3_encoders import H3EmbedConfig, H3RNNConfig, H3WeightedSumConfig
 from ludwig.utils import torch_utils
 
 logger = logging.getLogger(__name__)
@@ -197,6 +198,10 @@ def forward(self, inputs: torch.Tensor) -> Dict[str, torch.Tensor]:
 
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return H3EmbedConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([H3_INPUT_SIZE])
@@ -303,6 +308,10 @@ def forward(self, inputs: torch.Tensor) -> Dict[str, torch.Tensor]:
 
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return H3WeightedSumConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([H3_INPUT_SIZE])
@@ -431,6 +440,10 @@ def forward(self, inputs: torch.Tensor) -> Dict[str, torch.Tensor]:
 
         return {"encoder_output": hidden, "encoder_output_state": final_state}
 
+    @staticmethod
+    def get_schema_cls():
+        return H3RNNConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([H3_INPUT_SIZE])
diff --git a/ludwig/encoders/image_encoders.py b/ludwig/encoders/image_encoders.py
index 6a03e146581..4cc5f92471a 100644
--- a/ludwig/encoders/image_encoders.py
+++ b/ludwig/encoders/image_encoders.py
@@ -24,13 +24,19 @@
 from ludwig.modules.convolutional_modules import Conv2DStack, ResNet
 from ludwig.modules.fully_connected_modules import FCStack
 from ludwig.modules.mlp_mixer_modules import MLPMixer
+from ludwig.schema.encoders.image_encoders import (
+    MLPMixerEncoderConfig,
+    ResNetEncoderConfig,
+    Stacked2DCNNEncoderConfig,
+    ViTEncoderConfig,
+)
 from ludwig.utils.pytorch_utils import freeze_parameters
 
 logger = logging.getLogger(__name__)
 
 
 # TODO(shreya): Add type hints for missing args
-@register_encoder("stacked_cnn", IMAGE, default=True)
+@register_encoder("stacked_cnn", IMAGE)
 class Stacked2DCNN(Encoder):
     def __init__(
         self,
@@ -139,6 +145,10 @@ def forward(self, inputs: torch.Tensor) -> Dict[str, torch.Tensor]:
 
         return {"encoder_output": outputs}
 
+    @staticmethod
+    def get_schema_cls():
+        return Stacked2DCNNEncoderConfig
+
     @property
     def output_shape(self) -> torch.Size:
         return self.fc_stack.output_shape
@@ -223,6 +233,10 @@ def forward(self, inputs: torch.Tensor) -> Dict[str, torch.Tensor]:
         hidden = self.fc_stack(hidden)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return ResNetEncoderConfig
+
     @property
     def output_shape(self) -> torch.Size:
         return self.fc_stack.output_shape
@@ -280,6 +294,10 @@ def forward(self, inputs: torch.Tensor) -> Dict[str, torch.Tensor]:
         hidden = self.mlp_mixer(inputs)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return MLPMixerEncoderConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return self._input_shape
@@ -377,6 +395,10 @@ def forward(self, inputs: torch.Tensor, head_mask: torch.Tensor = None) -> Dict[
             return_dict["attentions"] = output.attentions
         return return_dict
 
+    @staticmethod
+    def get_schema_cls():
+        return ViTEncoderConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size(self._input_shape)
diff --git a/ludwig/encoders/registry.py b/ludwig/encoders/registry.py
index 09401127365..9cf30b037f7 100644
--- a/ludwig/encoders/registry.py
+++ b/ludwig/encoders/registry.py
@@ -1,7 +1,7 @@
 from typing import Dict, List, Type, Union
 
 from ludwig.encoders.base import Encoder
-from ludwig.utils.registry import DEFAULT_KEYS, Registry
+from ludwig.utils.registry import Registry
 
 encoder_registry = Registry()
 
@@ -16,7 +16,7 @@ def wrap(cls):
     return wrap
 
 
-def register_encoder(name: str, features: Union[str, List[str]], default=False):
+def register_encoder(name: str, features: Union[str, List[str]]):
     if isinstance(features, str):
         features = [features]
 
@@ -24,9 +24,6 @@ def wrap(cls):
         for feature in features:
             feature_registry = encoder_registry.get(feature, {})
             feature_registry[name] = cls
-            if default:
-                for key in DEFAULT_KEYS:
-                    feature_registry[key] = cls
             encoder_registry[feature] = feature_registry
         return cls
 
diff --git a/ludwig/encoders/sequence_encoders.py b/ludwig/encoders/sequence_encoders.py
index 49ddb9df0ce..e28be240e0f 100644
--- a/ludwig/encoders/sequence_encoders.py
+++ b/ludwig/encoders/sequence_encoders.py
@@ -28,11 +28,21 @@
 from ludwig.modules.fully_connected_modules import FCStack
 from ludwig.modules.recurrent_modules import RecurrentStack
 from ludwig.modules.reduction_modules import SequenceReducer
+from ludwig.schema.encoders.sequence_encoders import (
+    ParallelCNNConfig,
+    SequenceEmbedConfig,
+    SequencePassthroughConfig,
+    StackedCNNConfig,
+    StackedCNNRNNConfig,
+    StackedParallelCNNConfig,
+    StackedRNNConfig,
+    StackedTransformerConfig,
+)
 
 logger = logging.getLogger(__name__)
 
 
-@register_encoder("passthrough", [AUDIO, SEQUENCE, TEXT, TIMESERIES], default=True)
+@register_encoder("passthrough", [SEQUENCE, TEXT, TIMESERIES])
 class SequencePassthroughEncoder(Encoder):
     def __init__(self, reduce_output: str = None, max_sequence_length: int = 256, encoding_size: int = None, **kwargs):
         """
@@ -73,6 +83,10 @@ def forward(self, input_sequence, mask=None):
 
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return SequencePassthroughConfig
+
 
 @register_encoder("embed", [SEQUENCE, TEXT])
 class SequenceEmbedEncoder(Encoder):
@@ -195,6 +209,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None):
         hidden = self.reduce_sequence(embedded_sequence)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return SequenceEmbedConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -486,6 +504,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None):
 
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return ParallelCNNConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -785,6 +807,10 @@ def __init__(
                 default_dropout=dropout,
             )
 
+    @staticmethod
+    def get_schema_cls():
+        return StackedCNNConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1090,6 +1116,10 @@ def __init__(
                 default_dropout=dropout,
             )
 
+    @staticmethod
+    def get_schema_cls():
+        return StackedParallelCNNConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1355,6 +1385,10 @@ def __init__(
                 default_dropout=fc_dropout,
             )
 
+    @staticmethod
+    def get_schema_cls():
+        return StackedRNNConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1633,6 +1667,10 @@ def __init__(
                 default_dropout=fc_dropout,
             )
 
+    @staticmethod
+    def get_schema_cls():
+        return StackedCNNRNNConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1896,6 +1934,10 @@ def __init__(
                 default_dropout=fc_dropout,
             )
 
+    @staticmethod
+    def get_schema_cls():
+        return StackedTransformerConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
diff --git a/ludwig/encoders/set_encoders.py b/ludwig/encoders/set_encoders.py
index 6dc2612eb54..b6d90bee3e2 100644
--- a/ludwig/encoders/set_encoders.py
+++ b/ludwig/encoders/set_encoders.py
@@ -23,11 +23,12 @@
 from ludwig.encoders.registry import register_encoder
 from ludwig.modules.embedding_modules import EmbedSet
 from ludwig.modules.fully_connected_modules import FCStack
+from ludwig.schema.encoders.set_encoders import SetSparseEncoderConfig
 
 logger = logging.getLogger(__name__)
 
 
-@register_encoder("embed", SET, default=True)
+@register_encoder("embed", SET)
 class SetSparseEncoder(Encoder):
     def __init__(
         self,
@@ -96,6 +97,10 @@ def forward(self, inputs: torch.Tensor) -> torch.Tensor:
 
         return hidden
 
+    @staticmethod
+    def get_schema_cls():
+        return SetSparseEncoderConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.vocab_size])
diff --git a/ludwig/encoders/text_encoders.py b/ludwig/encoders/text_encoders.py
index 1f96c6a023e..eb929a6b74e 100644
--- a/ludwig/encoders/text_encoders.py
+++ b/ludwig/encoders/text_encoders.py
@@ -23,6 +23,26 @@
 from ludwig.encoders.base import Encoder
 from ludwig.encoders.registry import register_encoder
 from ludwig.modules.reduction_modules import SequenceReducer
+from ludwig.schema.encoders.text_encoders import (
+    ALBERTConfig,
+    AutoTransformerConfig,
+    BERTConfig,
+    CamemBERTConfig,
+    CTRLConfig,
+    DistilBERTConfig,
+    ELECTRAConfig,
+    FlauBERTConfig,
+    GPT2Config,
+    GPTConfig,
+    LongformerConfig,
+    MT5Config,
+    RoBERTaConfig,
+    T5Config,
+    TransformerXLConfig,
+    XLMConfig,
+    XLMRoBERTaConfig,
+    XLNetConfig,
+)
 from ludwig.utils.pytorch_utils import freeze_parameters
 
 logger = logging.getLogger(__name__)
@@ -135,6 +155,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
 
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return ALBERTConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -260,6 +284,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
 
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return MT5Config
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -357,6 +385,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
 
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return XLMRoBERTaConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -480,6 +512,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
 
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return BERTConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -619,6 +655,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return XLMConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -727,6 +767,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return GPTConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -831,6 +875,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return GPT2Config
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -917,6 +965,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
             hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return RoBERTaConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1039,6 +1091,10 @@ def forward(self, inputs: torch.Tensor, mask: torch.Tensor = None) -> Dict[str,
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return TransformerXLConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1170,6 +1226,10 @@ def forward(self, inputs: torch.Tensor, mask: torch.Tensor = None) -> Dict[str,
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return XLNetConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1274,6 +1334,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return DistilBERTConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1377,6 +1441,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return CTRLConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1407,7 +1475,7 @@ def __init__(
         self,
         max_sequence_length: int,
         use_pretrained: bool = True,
-        pretrained_model_name_or_path: str = "ctrl",
+        pretrained_model_name_or_path: str = "jplu/camembert-base",
         saved_weights_in_checkpoint: bool = False,
         reduce_output: str = "cls-pooled",
         trainable: bool = False,
@@ -1491,6 +1559,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
 
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return CamemBERTConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1598,6 +1670,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return T5Config
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1733,6 +1809,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return FlauBERTConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1846,6 +1926,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None) ->
         hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return ELECTRAConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -1934,6 +2018,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None):
             hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return LongformerConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
@@ -2014,6 +2102,10 @@ def forward(self, inputs: torch.Tensor, mask: Optional[torch.Tensor] = None):
             hidden = self.reduce_sequence(hidden, self.reduce_output)
         return {"encoder_output": hidden}
 
+    @staticmethod
+    def get_schema_cls():
+        return AutoTransformerConfig
+
     @property
     def input_shape(self) -> torch.Size:
         return torch.Size([self.max_sequence_length])
diff --git a/ludwig/features/audio_feature.py b/ludwig/features/audio_feature.py
index 8d6164e6d27..df0c77ebdc9 100644
--- a/ludwig/features/audio_feature.py
+++ b/ludwig/features/audio_feature.py
@@ -24,8 +24,8 @@
 from ludwig.constants import (
     AUDIO,
     AUDIO_FEATURE_KEYS,
-    BACKFILL,
     COLUMN,
+    ENCODER,
     NAME,
     PREPROCESSING,
     PROC_COLUMN,
@@ -56,6 +56,8 @@
 from ludwig.utils.misc_utils import set_default_value, set_default_values
 from ludwig.utils.types import TorchscriptPreprocessingInput
 
+logger = logging.getLogger(__name__)
+
 
 class _AudioPreprocessing(torch.nn.Module):
 
@@ -99,19 +101,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "audio_file_length_limit_in_s": 7.5,
-            "missing_value_strategy": BACKFILL,
-            "in_memory": True,
-            "padding_value": 0,
-            "norm": None,
-            "type": "fbank",
-            "window_length_in_s": 0.04,
-            "window_shift_in_s": 0.02,
-            "num_fft_points": None,
-            "window_type": "hamming",
-            "num_filter_bands": 80,
-        }
+        return AudioInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -232,7 +222,7 @@ def reduce(series):
             merged_stats["cropped"],
             audio_file_length_limit_in_s,
         )
-        logging.debug(print_statistics)
+        logger.debug(print_statistics)
         return processed_audio
 
     @staticmethod
@@ -242,7 +232,7 @@ def _transform_to_feature(
         audio_feature_dict: Dict[str, Union[float, int, str]],
         feature_dim: int,
         max_length: int,
-        padding_value: int,
+        padding_value: float,
         normalization_type: Optional[str] = None,
         type_key: str = TYPE,
     ):
@@ -371,7 +361,7 @@ def add_feature_data(
             raise ValueError("There are no audio files in the dataset provided.")
 
         first_audio_entry = next(iter(column))
-        logging.debug(f"Detected audio feature type is {type(first_audio_entry)}")
+        logger.debug(f"Detected audio feature type is {type(first_audio_entry)}")
 
         if not isinstance(first_audio_entry, str) and not isinstance(first_audio_entry, torch.Tensor):
             raise ValueError(
@@ -447,15 +437,13 @@ def _get_max_length_feature(preprocessing_parameters, sampling_rate_in_hz, audio
 
 @register_input_feature(AUDIO)
 class AudioInputFeature(AudioFeatureMixin, SequenceInputFeature):
-    encoder = "parallel_cnn"
-    max_sequence_length = None
-    embedding_size = None
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature, encoder_obj=encoder_obj)
-        if not self.embedding_size:
+    def __init__(self, input_feature_config: Union[AudioInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, encoder_obj=encoder_obj, **kwargs)
+        self.encoder_config = input_feature_config.encoder
+        if not getattr(self.encoder_config, "embedding_size", None):
             raise ValueError("embedding_size has to be defined - " 'check "update_config_with_metadata()"')
-        if not self.max_sequence_length:
+        if not getattr(self.encoder_config, "max_sequence_length", None):
             raise ValueError("max_sequence_length has to be defined - " 'check "update_config_with_metadata()"')
 
     def forward(self, inputs, mask=None):
@@ -469,7 +457,7 @@ def forward(self, inputs, mask=None):
 
     @property
     def input_shape(self) -> torch.Size:
-        return torch.Size([self.max_sequence_length, self.embedding_size])
+        return torch.Size([self.encoder_config.max_sequence_length, self.encoder_config.embedding_size])
 
     @property
     def input_dtype(self):
@@ -477,13 +465,16 @@ def input_dtype(self):
 
     @staticmethod
     def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs):
-        input_feature["max_sequence_length"] = feature_metadata["max_length"]
-        input_feature["embedding_size"] = feature_metadata["feature_dim"]
-        input_feature["should_embed"] = False
+        input_feature[ENCODER]["max_sequence_length"] = feature_metadata["max_length"]
+        input_feature[ENCODER]["embedding_size"] = feature_metadata["feature_dim"]
+        input_feature[ENCODER]["should_embed"] = False
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_values(input_feature, {TIED: None, "preprocessing": {}})
+        defaults = AudioInputFeatureConfig()
+        set_default_values(
+            input_feature, {ENCODER: {TYPE: defaults.encoder.type}, PREPROCESSING: {}, TIED: defaults.tied}
+        )
 
     @staticmethod
     def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
diff --git a/ludwig/features/bag_feature.py b/ludwig/features/bag_feature.py
index 31b137faa17..9619e78570e 100644
--- a/ludwig/features/bag_feature.py
+++ b/ludwig/features/bag_feature.py
@@ -15,19 +15,19 @@
 # ==============================================================================
 import logging
 from collections import Counter
-from typing import Any, Dict
+from typing import Any, Dict, Union
 
 import numpy as np
 import torch
 
-from ludwig.constants import BAG, COLUMN, FILL_WITH_CONST, NAME, PROC_COLUMN, TIED
+from ludwig.constants import BAG, COLUMN, ENCODER, NAME, PROC_COLUMN, TIED, TYPE
 from ludwig.features.base_feature import BaseFeatureMixin, InputFeature
 from ludwig.features.feature_utils import set_str_to_idx
 from ludwig.features.set_feature import _SetPreprocessing
 from ludwig.schema.features.bag_feature import BagInputFeatureConfig
 from ludwig.schema.features.utils import register_input_feature
-from ludwig.utils.misc_utils import set_default_value
-from ludwig.utils.strings_utils import create_vocabulary, UNKNOWN_SYMBOL
+from ludwig.utils.misc_utils import set_default_value, set_default_values
+from ludwig.utils.strings_utils import create_vocabulary
 
 logger = logging.getLogger(__name__)
 
@@ -39,13 +39,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "tokenizer": "space",
-            "most_common": 10000,
-            "lowercase": False,
-            "missing_value_strategy": FILL_WITH_CONST,
-            "fill_value": UNKNOWN_SYMBOL,
-        }
+        return BagInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -94,16 +88,14 @@ def add_feature_data(
 
 @register_input_feature(BAG)
 class BagInputFeature(BagFeatureMixin, InputFeature):
-    encoder = "embed"
-    vocab = []
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
+    def __init__(self, input_feature_config: Union[BagInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs):
         assert isinstance(inputs, torch.Tensor)
@@ -115,7 +107,7 @@ def forward(self, inputs):
 
     @property
     def input_shape(self) -> torch.Size:
-        return torch.Size([len(self.vocab)])
+        return torch.Size([len(self.encoder_config.vocab)])
 
     @property
     def output_shape(self) -> torch.Size:
@@ -123,11 +115,13 @@ def output_shape(self) -> torch.Size:
 
     @staticmethod
     def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs):
-        input_feature["vocab"] = feature_metadata["idx2str"]
+        input_feature[ENCODER]["vocab"] = feature_metadata["idx2str"]
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
+        defaults = BagInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def get_schema_cls():
diff --git a/ludwig/features/base_feature.py b/ludwig/features/base_feature.py
index 87cb202d111..4987f1bc142 100644
--- a/ludwig/features/base_feature.py
+++ b/ludwig/features/base_feature.py
@@ -12,15 +12,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
-import copy
 import logging
 from abc import ABC, abstractmethod, abstractstaticmethod
-from typing import Any, Dict, Optional
+from typing import Any, Dict, Optional, Union
 
 import torch
 from torch import Tensor
 
-from ludwig.constants import COLUMN, HIDDEN, LENGTHS, LOGITS, LOSS, NAME, PREDICTIONS, PROBABILITIES, PROC_COLUMN, TYPE
+from ludwig.constants import HIDDEN, LENGTHS, LOGITS, LOSS, PREDICTIONS, PROBABILITIES, TYPE
 from ludwig.decoders.registry import get_decoder_cls
 from ludwig.encoders.registry import get_encoder_cls
 from ludwig.features.feature_utils import compute_feature_hash, get_input_size_with_dependencies
@@ -29,10 +28,11 @@
 from ludwig.modules.metric_modules import MeanMetric
 from ludwig.modules.metric_registry import get_metric_classes, get_metric_cls
 from ludwig.modules.reduction_modules import SequenceReducer
+from ludwig.schema.features.base import BaseFeatureConfig, BaseOutputFeatureConfig
+from ludwig.schema.utils import assert_is_a_marshmallow_class
 from ludwig.utils import output_feature_utils
 from ludwig.utils.calibration import CalibrationModule
 from ludwig.utils.metric_utils import get_scalar_from_ludwig_metric
-from ludwig.utils.misc_utils import merge_dict
 from ludwig.utils.torch_utils import LudwigModule
 from ludwig.utils.types import DataFrame
 
@@ -122,39 +122,33 @@ class BaseFeature:
     classes to avoid the diamond pattern.
     """
 
-    def __init__(self, feature, *args, **kwargs):
+    def __init__(self, feature: BaseFeatureConfig):
         super().__init__()
 
-        if NAME not in feature:
+        if not feature.name:
             raise ValueError("Missing feature name")
-        self.feature_name = feature[NAME]
+        self.feature_name = feature.name
 
-        if COLUMN not in feature:
-            feature[COLUMN] = self.feature_name
-        self.column = feature[COLUMN]
+        if not feature.column:
+            feature.column = self.feature_name
+        self.column = feature.column
 
-        if PROC_COLUMN not in feature:
-            feature[PROC_COLUMN] = compute_feature_hash(feature)
-        self.proc_column = feature[PROC_COLUMN]
+        if not feature.proc_column:
+            feature.proc_column = compute_feature_hash(type(feature).Schema().dump(feature))
+        self.proc_column = feature.proc_column
 
-    def overwrite_defaults(self, feature):
-        attributes = set(self.__dict__.keys())
-        attributes.update(self.__class__.__dict__.keys())
-
-        for k in feature.keys():
-            if k in attributes:
-                if isinstance(feature[k], dict) and hasattr(self, k) and isinstance(getattr(self, k), dict):
-                    setattr(self, k, merge_dict(getattr(self, k), feature[k]))
-                else:
-                    setattr(self, k, feature[k])
+    @classmethod
+    def load_config(cls, feature: Union[BaseFeatureConfig, Dict]) -> BaseFeatureConfig:
+        if isinstance(feature, dict):
+            schema_cls = cls.get_schema_cls()
+            assert_is_a_marshmallow_class(schema_cls)
+            return schema_cls.Schema().load(feature)
+        return feature
 
 
 class InputFeature(BaseFeature, LudwigModule, ABC):
     """Parent class for all input features."""
 
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-
     def create_sample_input(self):
         # Used by get_model_inputs(), which is used for tracing-based torchscript generation.
         return torch.rand([2, *self.input_shape]).to(self.input_dtype)
@@ -169,8 +163,11 @@ def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs
     def populate_defaults(input_feature):
         pass
 
-    def initialize_encoder(self, encoder_parameters):
-        return get_encoder_cls(self.type(), self.encoder)(**encoder_parameters)
+    def initialize_encoder(self):
+        encoder_cls = get_encoder_cls(self.type(), self.encoder_config.type)
+        encoder_schema = encoder_cls.get_schema_cls().Schema()
+        encoder_params_dict = encoder_schema.dump(self.encoder_config)
+        return encoder_cls(**encoder_params_dict)
 
     @classmethod
     def get_preproc_input_dtype(cls, metadata: Dict[str, Any]) -> str:
@@ -184,7 +181,13 @@ def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
 class OutputFeature(BaseFeature, LudwigModule, ABC):
     """Parent class for all output features."""
 
-    def __init__(self, feature: Dict[str, Any], other_output_features: Dict[str, "OutputFeature"], *args, **kwargs):
+    def __init__(
+        self,
+        feature: BaseOutputFeatureConfig,
+        other_output_features: Dict[str, "OutputFeature"],
+        *args,
+        **kwargs,
+    ):
         """Defines defaults, overwrites them based on the feature dictionary, and sets up dependencies.
 
         Any output feature can depend on one or more other output features. The `other_output_features` input dictionary
@@ -192,48 +195,37 @@ def __init__(self, feature: Dict[str, Any], other_output_features: Dict[str, "Ou
         in topographically sorted order. Attributes of any dependent output features are used to properly initialize
         this feature's sizes.
         """
-        super().__init__(*args, feature=feature, **kwargs)
+        super().__init__(feature)
 
-        self.reduce_input = None
-        self.reduce_dependencies = None
+        self.decoder_config = feature.decoder
+        self.loss = feature.loss
+        self.reduce_input = feature.reduce_input
+        self.reduce_dependencies = feature.reduce_dependencies
 
         # List of feature names that this output feature is dependent on.
-        self.dependencies = []
-
-        self.fc_layers = None
-        self.num_fc_layers = 0
-        self.output_size = 256
-        self.use_bias = True
-        self.weights_initializer = "xavier_uniform"
-        self.bias_initializer = "zeros"
-        self.norm = None
-        self.norm_params = None
-        self.activation = "relu"
-        self.dropout = 0
-        self.input_size = None
-
-        self.overwrite_defaults(feature)
+        self.dependencies = feature.dependencies
 
         logger.debug(" output feature fully connected layers")
         logger.debug("  FCStack")
 
-        self.input_size = get_input_size_with_dependencies(self.input_size, self.dependencies, other_output_features)
-        feature["input_size"] = self.input_size  # needed for future overrides
+        feature.decoder.input_size = get_input_size_with_dependencies(
+            feature.decoder.input_size, self.dependencies, other_output_features
+        )
 
         self.fc_stack = FCStack(
-            first_layer_input_size=self.input_size,
-            layers=self.fc_layers,
-            num_layers=self.num_fc_layers,
-            default_output_size=self.output_size,
-            default_use_bias=self.use_bias,
-            default_weights_initializer=self.weights_initializer,
-            default_bias_initializer=self.bias_initializer,
-            default_norm=self.norm,
-            default_norm_params=self.norm_params,
-            default_activation=self.activation,
-            default_dropout=self.dropout,
+            first_layer_input_size=feature.decoder.input_size,
+            layers=feature.decoder.fc_layers,
+            num_layers=feature.decoder.num_fc_layers,
+            default_output_size=feature.decoder.output_size,
+            default_use_bias=feature.decoder.use_bias,
+            default_weights_initializer=feature.decoder.weights_initializer,
+            default_bias_initializer=feature.decoder.bias_initializer,
+            default_norm=feature.decoder.norm,
+            default_norm_params=feature.decoder.norm_params,
+            default_activation=feature.decoder.activation,
+            default_dropout=feature.decoder.dropout,
         )
-        self._calibration_module = self.create_calibration_module(feature)
+        self._calibration_module = self.create_calibration_module(kwargs)
         self._prediction_module = self.create_predict_module()
 
         # set up two sequence reducers, one for inputs and other for dependencies
@@ -264,15 +256,11 @@ def get_output_dtype(cls):
     def metric_functions(self) -> Dict:
         pass
 
-    def initialize_decoder(self, decoder_parameters):
-        decoder_parameters_copy = copy.copy(decoder_parameters)
+    def initialize_decoder(self):
         # Input to the decoder is the output feature's FC hidden layer.
-        decoder_parameters_copy["input_size"] = self.fc_stack.output_shape[-1]
-        if "decoder" in decoder_parameters:
-            decoder = decoder_parameters["decoder"]
-        else:
-            decoder = self.decoder
-        return get_decoder_cls(self.type(), decoder)(**decoder_parameters_copy)
+        self.decoder_config.input_size = self.fc_stack.output_shape[-1]
+        decoder = self.decoder_config.type
+        return get_decoder_cls(self.type(), decoder)(**self.decoder_config.__dict__)
 
     def train_loss(self, targets: Tensor, predictions: Dict[str, Tensor], feature_name):
         loss_class = type(self.train_loss_function)
diff --git a/ludwig/features/binary_feature.py b/ludwig/features/binary_feature.py
index 46beb668ac0..9f51c5c2c99 100644
--- a/ludwig/features/binary_feature.py
+++ b/ludwig/features/binary_feature.py
@@ -14,7 +14,7 @@
 # limitations under the License.
 # ==============================================================================
 import logging
-from typing import Any, Dict, List, Tuple
+from typing import Any, Dict, List, Tuple, Union
 
 import numpy as np
 import torch
@@ -22,9 +22,10 @@
 from ludwig.constants import (
     ACCURACY,
     BINARY,
-    BINARY_WEIGHTED_CROSS_ENTROPY,
     COLUMN,
-    FILL_WITH_FALSE,
+    DECODER,
+    DEPENDENCIES,
+    ENCODER,
     HIDDEN,
     LOGITS,
     LOSS,
@@ -33,8 +34,10 @@
     PROBABILITIES,
     PROBABILITY,
     PROC_COLUMN,
+    REDUCE_DEPENDENCIES,
+    REDUCE_INPUT,
     ROC_AUC,
-    SUM,
+    THRESHOLD,
     TIED,
     TYPE,
 )
@@ -132,9 +135,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults() -> Dict[str, Any]:
-        return {
-            "missing_value_strategy": FILL_WITH_FALSE,
-        }
+        return BinaryInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -214,17 +215,14 @@ def add_feature_data(
 
 @register_input_feature(BINARY)
 class BinaryInputFeature(BinaryFeatureMixin, InputFeature):
-    encoder = "passthrough"
-    norm = None
-    dropout = False
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
+    def __init__(self, input_feature_config: Union[BinaryInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs):
         assert isinstance(inputs, torch.Tensor)
@@ -254,7 +252,9 @@ def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
+        defaults = BinaryInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def get_schema_cls():
@@ -274,16 +274,18 @@ def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
 
 @register_output_feature(BINARY)
 class BinaryOutputFeature(BinaryFeatureMixin, OutputFeature):
-    decoder = "regressor"
-    loss = {TYPE: BINARY_WEIGHTED_CROSS_ENTROPY}
     metric_functions = {LOSS: None, ACCURACY: None, ROC_AUC: None}
     default_validation_metric = ROC_AUC
-    threshold = 0.5
 
-    def __init__(self, feature, output_features: Dict[str, OutputFeature]):
-        super().__init__(feature, output_features)
-        self.overwrite_defaults(feature)
-        self.decoder_obj = self.initialize_decoder(feature)
+    def __init__(
+        self,
+        output_feature_config: Union[BinaryOutputFeatureConfig, Dict],
+        output_features: Dict[str, OutputFeature],
+        **kwargs,
+    ):
+        output_feature_config = self.load_config(output_feature_config)
+        super().__init__(output_feature_config, output_features, **kwargs)
+        self.decoder_obj = self.initialize_decoder()
         self._setup_loss()
         self._setup_metrics()
 
@@ -310,7 +312,10 @@ def create_calibration_module(self, feature) -> torch.nn.Module:
         return None
 
     def create_predict_module(self) -> PredictModule:
-        return _BinaryPredict(self.threshold, calibration_module=self.calibration_module)
+        # A lot of code assumes output features have a prediction module, but if we are using GBM then passthrough
+        # decoder is specified here which has no threshold.
+        threshold = getattr(self.decoder_config, "threshold", 0.5)
+        return _BinaryPredict(threshold, calibration_module=self.calibration_module)
 
     def get_prediction_set(self):
         return {PREDICTIONS, PROBABILITIES, LOGITS}
@@ -399,25 +404,22 @@ def postprocess_predictions(
 
     @staticmethod
     def populate_defaults(output_feature):
+        defaults = BinaryOutputFeatureConfig()
+
         # If Loss is not defined, set an empty dictionary
         set_default_value(output_feature, LOSS, {})
-        set_default_values(
-            output_feature[LOSS],
-            {
-                "robust_lambda": 0,
-                "confidence_penalty": 0,
-                "positive_class_weight": None,  # Weight for each label.
-                "weight": 1,  # Weight across output features.
-            },
-        )
+        set_default_values(output_feature[LOSS], defaults.loss)
 
         set_default_values(
             output_feature,
             {
-                "threshold": 0.5,
-                "dependencies": [],
-                "reduce_input": SUM,
-                "reduce_dependencies": SUM,
+                DECODER: {
+                    TYPE: defaults.decoder.type,
+                    THRESHOLD: defaults.decoder.threshold,
+                },
+                DEPENDENCIES: defaults.dependencies,
+                REDUCE_INPUT: defaults.reduce_input,
+                REDUCE_DEPENDENCIES: defaults.reduce_dependencies,
             },
         )
 
diff --git a/ludwig/features/category_feature.py b/ludwig/features/category_feature.py
index 8dd4dd56afa..f5a1550de85 100644
--- a/ludwig/features/category_feature.py
+++ b/ludwig/features/category_feature.py
@@ -14,7 +14,7 @@
 # limitations under the License.
 # ==============================================================================
 import logging
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Union
 
 import numpy as np
 import torch
@@ -23,7 +23,9 @@
     ACCURACY,
     CATEGORY,
     COLUMN,
-    FILL_WITH_CONST,
+    DECODER,
+    DEPENDENCIES,
+    ENCODER,
     HIDDEN,
     HITS_AT_K,
     LOGITS,
@@ -34,9 +36,10 @@
     PROBABILITY,
     PROC_COLUMN,
     PROJECTION_INPUT,
-    SOFTMAX_CROSS_ENTROPY,
-    SUM,
+    REDUCE_DEPENDENCIES,
+    REDUCE_INPUT,
     TIED,
+    TOP_K,
     TYPE,
 )
 from ludwig.features.base_feature import BaseFeatureMixin, InputFeature, OutputFeature, PredictModule
@@ -116,12 +119,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "most_common": 10000,
-            "lowercase": False,
-            "missing_value_strategy": FILL_WITH_CONST,
-            "fill_value": UNKNOWN_SYMBOL,
-        }
+        return CategoryInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -161,15 +159,14 @@ def add_feature_data(
 
 @register_input_feature(CATEGORY)
 class CategoryInputFeature(CategoryFeatureMixin, InputFeature):
-    encoder = "dense"
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
+    def __init__(self, input_feature_config: Union[CategoryInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs):
         assert isinstance(inputs, torch.Tensor)
@@ -204,11 +201,13 @@ def output_shape(self) -> torch.Size:
 
     @staticmethod
     def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs):
-        input_feature["vocab"] = feature_metadata["idx2str"]
+        input_feature[ENCODER]["vocab"] = feature_metadata["idx2str"]
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
+        defaults = CategoryInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def get_schema_cls():
@@ -221,17 +220,19 @@ def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
 
 @register_output_feature(CATEGORY)
 class CategoryOutputFeature(CategoryFeatureMixin, OutputFeature):
-    decoder = "classifier"
-    loss = {TYPE: SOFTMAX_CROSS_ENTROPY}
     metric_functions = {LOSS: None, ACCURACY: None, HITS_AT_K: None}
     default_validation_metric = ACCURACY
-    num_classes = 0
-    top_k = 3
 
-    def __init__(self, feature, output_features: Dict[str, OutputFeature]):
-        super().__init__(feature, output_features)
-        self.overwrite_defaults(feature)
-        self.decoder_obj = self.initialize_decoder(feature)
+    def __init__(
+        self,
+        output_feature_config: Union[CategoryOutputFeatureConfig, Dict],
+        output_features: Dict[str, OutputFeature],
+        **kwargs,
+    ):
+        output_feature_config = self.load_config(output_feature_config)
+        super().__init__(output_feature_config, output_features, **kwargs)
+        self.top_k = output_feature_config.top_k
+        self.decoder_obj = self.initialize_decoder()
         self._setup_loss()
         self._setup_metrics()
 
@@ -251,7 +252,7 @@ def create_calibration_module(self, feature) -> torch.nn.Module:
         """
         if feature.get("calibration"):
             calibration_cls = calibration.get_calibration_cls(CATEGORY, "temperature_scaling")
-            return calibration_cls(num_classes=feature["num_classes"])
+            return calibration_cls(num_classes=self.decoder_config.num_classes)
         return None
 
     def create_predict_module(self) -> PredictModule:
@@ -277,11 +278,11 @@ def metric_kwargs(self):
 
     @staticmethod
     def update_config_with_metadata(output_feature, feature_metadata, *args, **kwargs):
-        output_feature["num_classes"] = feature_metadata["vocab_size"]
-        output_feature["top_k"] = min(output_feature["num_classes"], output_feature["top_k"])
+        output_feature[DECODER]["num_classes"] = feature_metadata["vocab_size"]
+        output_feature["top_k"] = min(output_feature[DECODER]["num_classes"], output_feature["top_k"])
 
         if isinstance(output_feature[LOSS]["class_weights"], (list, tuple)):
-            if len(output_feature[LOSS]["class_weights"]) != output_feature["num_classes"]:
+            if len(output_feature[LOSS]["class_weights"]) != output_feature[DECODER]["num_classes"]:
                 raise ValueError(
                     "The length of class_weights ({}) is not compatible with "
                     "the number of classes ({}) for feature {}. "
@@ -289,7 +290,7 @@ def update_config_with_metadata(output_feature, feature_metadata, *args, **kwarg
                     "and their order and consider there needs to be a weight "
                     "for the <UNK> class too.".format(
                         len(output_feature[LOSS]["class_weights"]),
-                        output_feature["num_classes"],
+                        output_feature[DECODER]["num_classes"],
                         output_feature[COLUMN],
                     )
                 )
@@ -419,26 +420,26 @@ def postprocess_predictions(
 
     @staticmethod
     def populate_defaults(output_feature):
+        defaults = CategoryOutputFeatureConfig()
+
         # If Loss is not defined, set an empty dictionary
         set_default_value(output_feature, LOSS, {})
-
         # Populate the default values for LOSS if they aren't defined already
+        set_default_values(output_feature[LOSS], defaults.loss)
+
         set_default_values(
-            output_feature[LOSS],
+            output_feature,
             {
-                TYPE: "softmax_cross_entropy",
-                "class_weights": 1,
-                "robust_lambda": 0,
-                "confidence_penalty": 0,
-                "class_similarities_temperature": 0,
-                "weight": 1,
+                DECODER: {
+                    TYPE: defaults.decoder.type,
+                },
+                TOP_K: defaults.top_k,
+                DEPENDENCIES: defaults.dependencies,
+                REDUCE_INPUT: defaults.reduce_input,
+                REDUCE_DEPENDENCIES: defaults.reduce_dependencies,
             },
         )
 
-        set_default_values(
-            output_feature, {"top_k": 3, "dependencies": [], "reduce_input": SUM, "reduce_dependencies": SUM}
-        )
-
     @staticmethod
     def get_schema_cls():
         return CategoryOutputFeatureConfig
diff --git a/ludwig/features/date_feature.py b/ludwig/features/date_feature.py
index 57cfb24c336..3646a96af88 100644
--- a/ludwig/features/date_feature.py
+++ b/ludwig/features/date_feature.py
@@ -15,18 +15,18 @@
 # ==============================================================================
 import logging
 from datetime import datetime
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Union
 
 import numpy as np
 import torch
 from dateutil.parser import parse
 
-from ludwig.constants import COLUMN, DATE, FILL_WITH_CONST, PROC_COLUMN, TIED
+from ludwig.constants import COLUMN, DATE, ENCODER, PROC_COLUMN, TIED, TYPE
 from ludwig.features.base_feature import BaseFeatureMixin, InputFeature
 from ludwig.schema.features.date_feature import DateInputFeatureConfig
 from ludwig.schema.features.utils import register_input_feature
 from ludwig.utils.date_utils import create_vector_from_datetime_obj
-from ludwig.utils.misc_utils import set_default_value
+from ludwig.utils.misc_utils import set_default_value, set_default_values
 from ludwig.utils.types import DataFrame, TorchscriptPreprocessingInput
 
 logger = logging.getLogger(__name__)
@@ -55,7 +55,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {"missing_value_strategy": FILL_WITH_CONST, "fill_value": "", "datetime_format": None}
+        return DateInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -75,7 +75,7 @@ def date_to_list(date_str, datetime_format, preprocessing_parameters):
             else:
                 datetime_obj = parse(date_str)
         except Exception as e:
-            logging.error(
+            logger.error(
                 f"Error parsing date: {date_str} with error {e} "
                 "Please provide a datetime format that parses it "
                 "in the preprocessing section of the date feature "
@@ -114,15 +114,14 @@ def add_feature_data(
 
 @register_input_feature(DATE)
 class DateInputFeature(DateFeatureMixin, InputFeature):
-    encoder = "embed"
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
+    def __init__(self, input_feature_config: Union[DateInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs):
         assert isinstance(inputs, torch.Tensor)
@@ -151,7 +150,9 @@ def create_sample_input(self):
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
+        defaults = DateInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def get_schema_cls():
diff --git a/ludwig/features/feature_utils.py b/ludwig/features/feature_utils.py
index a21c2b11509..bc606a42364 100644
--- a/ludwig/features/feature_utils.py
+++ b/ludwig/features/feature_utils.py
@@ -116,11 +116,11 @@ def get_input_size_with_dependencies(
     """
     input_size_with_dependencies = combiner_output_size
     for feature_name in dependencies:
-        if other_output_features[feature_name].num_fc_layers:
+        if other_output_features[feature_name].decoder_config.num_fc_layers:
             input_size_with_dependencies += other_output_features[feature_name].fc_stack.output_shape[-1]
         else:
             # 0-layer FCStack. Use the output feature's input size.
-            input_size_with_dependencies += other_output_features[feature_name].input_size
+            input_size_with_dependencies += other_output_features[feature_name].decoder_config.input_size
     return input_size_with_dependencies
 
 
diff --git a/ludwig/features/h3_feature.py b/ludwig/features/h3_feature.py
index 8ba07f8aced..ab7d1323d60 100644
--- a/ludwig/features/h3_feature.py
+++ b/ludwig/features/h3_feature.py
@@ -1,4 +1,3 @@
-#! /usr/bin/env python
 # Copyright (c) 2019 Uber Technologies, Inc.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -14,17 +13,17 @@
 # limitations under the License.
 # ==============================================================================
 import logging
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Union
 
 import numpy as np
 import torch
 
-from ludwig.constants import COLUMN, FILL_WITH_CONST, H3, PROC_COLUMN, TIED
+from ludwig.constants import COLUMN, ENCODER, H3, PROC_COLUMN, TIED, TYPE
 from ludwig.features.base_feature import BaseFeatureMixin, InputFeature
 from ludwig.schema.features.h3_feature import H3InputFeatureConfig
 from ludwig.schema.features.utils import register_input_feature
 from ludwig.utils.h3_util import h3_to_components
-from ludwig.utils.misc_utils import set_default_value
+from ludwig.utils.misc_utils import set_default_value, set_default_values
 from ludwig.utils.types import TorchscriptPreprocessingInput
 
 logger = logging.getLogger(__name__)
@@ -74,11 +73,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "missing_value_strategy": FILL_WITH_CONST,
-            "fill_value": 576495936675512319
-            # mode 1 edge 0 resolution 0 base_cell 0
-        }
+        return H3InputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -116,15 +111,14 @@ def add_feature_data(
 
 @register_input_feature(H3)
 class H3InputFeature(H3FeatureMixin, InputFeature):
-    encoder = "embed"
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
+    def __init__(self, input_feature_config: Union[H3InputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs):
         assert isinstance(inputs, torch.Tensor)
@@ -153,7 +147,9 @@ def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
+        defaults = H3InputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
diff --git a/ludwig/features/image_feature.py b/ludwig/features/image_feature.py
index ca91e5762f9..2947217b604 100644
--- a/ludwig/features/image_feature.py
+++ b/ludwig/features/image_feature.py
@@ -25,9 +25,9 @@
 import torchvision
 
 from ludwig.constants import (
-    BACKFILL,
     CHECKSUM,
     COLUMN,
+    ENCODER,
     HEIGHT,
     IMAGE,
     INFER_IMAGE_DIMENSIONS,
@@ -41,6 +41,7 @@
     SRC,
     TIED,
     TRAINING,
+    TYPE,
     WIDTH,
 )
 from ludwig.data.cache.types import wrap
@@ -58,9 +59,12 @@
     read_image_from_path,
     resize_image,
 )
-from ludwig.utils.misc_utils import set_default_value
+from ludwig.utils.misc_utils import set_default_value, set_default_values
 from ludwig.utils.types import Series, TorchscriptPreprocessingInput
 
+logger = logging.getLogger(__name__)
+
+
 # TODO(shreya): Confirm if it's ok to do per channel normalization
 # TODO(shreya): Also confirm if this is being used anywhere
 # TODO(shreya): Confirm if ok to use imagenet means and std devs
@@ -127,18 +131,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "missing_value_strategy": BACKFILL,
-            "in_memory": True,
-            "resize_method": "interpolate",
-            "scaling": "pixel_normalization",
-            "num_processes": 1,
-            "infer_image_num_channels": True,
-            "infer_image_dimensions": True,
-            "infer_image_max_height": 256,
-            "infer_image_max_width": 256,
-            "infer_image_sample_size": 100,
-        }
+        return ImageInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -259,7 +252,7 @@ def _infer_image_size(image_sample: List[torch.Tensor], max_height: int, max_wid
         height = min(int(round(height_avg)), max_height)
         width = min(int(round(width_avg)), max_width)
 
-        logging.debug(f"Inferring height: {height} and width: {width}")
+        logger.debug(f"Inferring height: {height} and width: {width}")
         return height, width
 
     @staticmethod
@@ -311,7 +304,7 @@ def _finalize_preprocessing_parameters(
         expected be of the same size with the same number of channels
         """
 
-        explicit_height_width = HEIGHT in preprocessing_parameters or WIDTH in preprocessing_parameters
+        explicit_height_width = preprocessing_parameters[HEIGHT] or preprocessing_parameters[WIDTH]
         explicit_num_channels = NUM_CHANNELS in preprocessing_parameters and preprocessing_parameters[NUM_CHANNELS]
 
         if preprocessing_parameters[INFER_IMAGE_DIMENSIONS] and not (explicit_height_width and explicit_num_channels):
@@ -488,19 +481,14 @@ def add_feature_data(
 
 @register_input_feature(IMAGE)
 class ImageInputFeature(ImageFeatureMixin, InputFeature):
-    height = 0
-    width = 0
-    num_channels = 0
-    scaling = "pixel_normalization"
-    encoder = "stacked_cnn"
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
+    def __init__(self, input_feature_config: Union[ImageInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs: torch.Tensor) -> torch.Tensor:
         assert isinstance(inputs, torch.Tensor)
@@ -519,7 +507,7 @@ def input_dtype(self):
 
     @property
     def input_shape(self) -> torch.Size:
-        return torch.Size([self.num_channels, self.height, self.width])
+        return torch.Size([self.encoder_config.num_channels, self.encoder_config.height, self.encoder_config.width])
 
     @property
     def output_shape(self) -> torch.Size:
@@ -528,12 +516,14 @@ def output_shape(self) -> torch.Size:
     @staticmethod
     def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs):
         for key in ["height", "width", "num_channels", "scaling"]:
-            input_feature[key] = feature_metadata[PREPROCESSING][key]
+            input_feature[ENCODER][key] = feature_metadata[PREPROCESSING][key]
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
+        defaults = ImageInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
         set_default_value(input_feature, PREPROCESSING, {})
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def get_schema_cls():
diff --git a/ludwig/features/number_feature.py b/ludwig/features/number_feature.py
index c70686a6eb0..4ae7eefc052 100644
--- a/ludwig/features/number_feature.py
+++ b/ludwig/features/number_feature.py
@@ -15,7 +15,7 @@
 # ==============================================================================
 import logging
 import random
-from typing import Any, Dict
+from typing import Any, Dict, Union
 
 import numpy as np
 import pandas as pd
@@ -23,8 +23,11 @@
 from torch import nn
 
 from ludwig.constants import (
+    CLIP,
     COLUMN,
-    FILL_WITH_CONST,
+    DECODER,
+    DEPENDENCIES,
+    ENCODER,
     HIDDEN,
     LOGITS,
     LOSS,
@@ -35,9 +38,10 @@
     PREDICTIONS,
     PROC_COLUMN,
     R2,
+    REDUCE_DEPENDENCIES,
+    REDUCE_INPUT,
     ROOT_MEAN_SQUARED_ERROR,
     ROOT_MEAN_SQUARED_PERCENTAGE_ERROR,
-    SUM,
     TIED,
     TYPE,
 )
@@ -225,11 +229,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "missing_value_strategy": FILL_WITH_CONST,
-            "fill_value": 0,
-            "normalization": None,
-        }
+        return NumberInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -283,17 +283,15 @@ def normalize(series: pd.Series) -> pd.Series:
 
 @register_input_feature(NUMBER)
 class NumberInputFeature(NumberFeatureMixin, InputFeature):
-    encoder = "passthrough"
-
-    def __init__(self, feature, encoder_obj=None):
-        # Required for certain encoders, maybe pass into initialize_encoder
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
-        feature["input_size"] = self.input_shape[-1]
+    def __init__(self, input_feature_config: Union[NumberInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
+        self.encoder_config.input_size = self.input_shape[-1]
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs):
         assert isinstance(inputs, torch.Tensor)
@@ -324,7 +322,9 @@ def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
+        defaults = NumberInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def get_schema_cls():
@@ -341,8 +341,6 @@ def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
 
 @register_output_feature(NUMBER)
 class NumberOutputFeature(NumberFeatureMixin, OutputFeature):
-    decoder = "regressor"
-    loss = {TYPE: MEAN_SQUARED_ERROR}
     metric_functions = {
         LOSS: None,
         MEAN_SQUARED_ERROR: None,
@@ -352,12 +350,16 @@ class NumberOutputFeature(NumberFeatureMixin, OutputFeature):
         R2: None,
     }
     default_validation_metric = MEAN_SQUARED_ERROR
-    clip = None
 
-    def __init__(self, feature, output_features: Dict[str, OutputFeature]):
-        super().__init__(feature, output_features)
-        self.overwrite_defaults(feature)
-        self.decoder_obj = self.initialize_decoder(feature)
+    def __init__(
+        self,
+        output_feature_config: Union[NumberOutputFeatureConfig, Dict],
+        output_features: Dict[str, OutputFeature],
+        **kwargs,
+    ):
+        output_feature_config = self.load_config(output_feature_config)
+        super().__init__(output_feature_config, output_features, **kwargs)
+        self.decoder_obj = self.initialize_decoder()
         self._setup_loss()
         self._setup_metrics()
 
@@ -366,19 +368,21 @@ def logits(self, inputs, **kwargs):  # hidden
         return self.decoder_obj(hidden)
 
     def create_predict_module(self) -> PredictModule:
-        if self.clip is not None and not (isinstance(self.clip, (list, tuple)) and len(self.clip) == 2):
+        if getattr(self.decoder_config, "clip", None) and not (
+            isinstance(self.decoder_config.clip, (list, tuple)) and len(self.decoder_config.clip) == 2
+        ):
             raise ValueError(
                 f"The clip parameter of {self.feature_name} is {self.clip}. "
                 f"It must be a list or a tuple of length 2."
             )
-        return _NumberPredict(self.clip)
+        return _NumberPredict(getattr(self.decoder_config, "clip", None))
 
     def get_prediction_set(self):
         return {PREDICTIONS, LOGITS}
 
     @property
     def input_shape(self) -> torch.Size:
-        return torch.Size([self.input_size])
+        return torch.Size([self.decoder_config.input_size])
 
     @classmethod
     def get_output_dtype(cls):
@@ -417,17 +421,19 @@ def postprocess_predictions(
 
     @staticmethod
     def populate_defaults(output_feature):
-        set_default_value(output_feature, LOSS, {TYPE: "mean_squared_error", "weight": 1})
-        set_default_value(output_feature[LOSS], TYPE, "mean_squared_error")
-        set_default_value(output_feature[LOSS], "weight", 1)
-
+        defaults = NumberOutputFeatureConfig()
+        set_default_value(output_feature, LOSS, {})
+        set_default_values(output_feature[LOSS], defaults.loss)
         set_default_values(
             output_feature,
             {
-                "clip": None,
-                "dependencies": [],
-                "reduce_input": SUM,
-                "reduce_dependencies": SUM,
+                DECODER: {
+                    TYPE: defaults.decoder.type,
+                    CLIP: defaults.decoder.clip,
+                },
+                DEPENDENCIES: defaults.dependencies,
+                REDUCE_INPUT: defaults.reduce_input,
+                REDUCE_DEPENDENCIES: defaults.reduce_dependencies,
             },
         )
 
diff --git a/ludwig/features/sequence_feature.py b/ludwig/features/sequence_feature.py
index 00d3dd54e71..517ca277948 100644
--- a/ludwig/features/sequence_feature.py
+++ b/ludwig/features/sequence_feature.py
@@ -16,15 +16,17 @@
 
 import logging
 from functools import partial
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Union
 
 import numpy as np
 import torch
 
 from ludwig.constants import (
     COLUMN,
+    DECODER,
+    DEPENDENCIES,
     EDIT_DISTANCE,
-    FILL_WITH_CONST,
+    ENCODER,
     LAST_ACCURACY,
     LAST_PREDICTIONS,
     LENGTHS,
@@ -35,9 +37,10 @@
     PROBABILITIES,
     PROBABILITY,
     PROC_COLUMN,
+    REDUCE_DEPENDENCIES,
+    REDUCE_INPUT,
     SEQUENCE,
     SEQUENCE_ACCURACY,
-    SUM,
     TIED,
     TOKEN_ACCURACY,
     TYPE,
@@ -48,11 +51,10 @@
 from ludwig.schema.features.utils import register_input_feature, register_output_feature
 from ludwig.utils import output_feature_utils
 from ludwig.utils.math_utils import softmax
-from ludwig.utils.misc_utils import get_from_registry, set_default_value
+from ludwig.utils.misc_utils import get_from_registry, set_default_value, set_default_values
 from ludwig.utils.strings_utils import (
     build_sequence_matrix,
     create_vocabulary,
-    PADDING_SYMBOL,
     SpecialSymbol,
     START_SYMBOL,
     STOP_SYMBOL,
@@ -200,18 +202,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "max_sequence_length": 256,
-            "most_common": 20000,
-            "padding_symbol": PADDING_SYMBOL,
-            "unknown_symbol": UNKNOWN_SYMBOL,
-            "padding": "right",
-            "tokenizer": "space",
-            "lowercase": False,
-            "vocab_file": None,
-            "missing_value_strategy": FILL_WITH_CONST,
-            "fill_value": UNKNOWN_SYMBOL,
-        }
+        return SequenceInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -270,19 +261,17 @@ def add_feature_data(
 
 @register_input_feature(SEQUENCE)
 class SequenceInputFeature(SequenceFeatureMixin, InputFeature):
-    encoder = "parallel_cnn"
-    max_sequence_length = None
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
+    def __init__(self, input_feature_config: Union[SequenceInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
         # TODO: Potentially abstract this feature-specific attribute overwrite to a consolidated design.
-        if "vocab" in feature:
-            feature["vocab_size"] = len(feature["vocab"])
-        self.overwrite_defaults(feature)
+        self.encoder_config = input_feature_config.encoder
+        if getattr(self.encoder_config, "vocab", None):
+            self.encoder_config.vocab_size = len(self.encoder_config.vocab)
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs: torch.Tensor, mask=None):
         assert isinstance(inputs, torch.Tensor)
@@ -301,13 +290,14 @@ def input_dtype(self):
 
     @staticmethod
     def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs):
-        input_feature["vocab"] = feature_metadata["idx2str"]
-        input_feature["max_sequence_length"] = feature_metadata["max_sequence_length"]
+        input_feature[ENCODER]["vocab"] = feature_metadata["idx2str"]
+        input_feature[ENCODER]["max_sequence_length"] = feature_metadata["max_sequence_length"]
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
-        set_default_value(input_feature, "encoder", "parallel_cnn")
+        defaults = SequenceInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def get_schema_cls():
@@ -315,7 +305,7 @@ def get_schema_cls():
 
     @property
     def input_shape(self) -> torch.Size:
-        return torch.Size([self.max_sequence_length])
+        return torch.Size([self.encoder_config.max_sequence_length])
 
     @property
     def output_shape(self) -> torch.Size:
@@ -328,8 +318,6 @@ def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
 
 @register_output_feature(SEQUENCE)
 class SequenceOutputFeature(SequenceFeatureMixin, OutputFeature):
-    decoder = "generator"
-    loss = {TYPE: "sequence_softmax_cross_entropy"}
     metric_functions = {
         LOSS: None,
         TOKEN_ACCURACY: None,
@@ -339,13 +327,16 @@ class SequenceOutputFeature(SequenceFeatureMixin, OutputFeature):
         EDIT_DISTANCE: None,
     }
     default_validation_metric = LOSS
-    max_sequence_length = 0
-    num_classes = 0
 
-    def __init__(self, feature: Dict[str, Any], output_features: Dict[str, OutputFeature]):
-        super().__init__(feature, output_features)
-        self.overwrite_defaults(feature)
-        self.decoder_obj = self.initialize_decoder(feature)
+    def __init__(
+        self,
+        output_feature_config: Union[SequenceOutputFeatureConfig, Dict],
+        output_features: Dict[str, OutputFeature],
+        **kwargs,
+    ):
+        output_feature_config = self.load_config(output_feature_config)
+        super().__init__(output_feature_config, output_features, **kwargs)
+        self.decoder_obj = self.initialize_decoder()
         self._setup_loss()
         self._setup_metrics()
 
@@ -369,14 +360,14 @@ def input_shape(self) -> torch.Size:
 
     @property
     def output_shape(self) -> torch.Size:
-        return torch.Size([self.max_sequence_length])
+        return torch.Size([self.decoder_config.max_sequence_length])
 
     @staticmethod
     def update_config_with_metadata(output_feature, feature_metadata, *args, **kwargs):
-        output_feature["vocab_size"] = feature_metadata["vocab_size"]
-        output_feature["max_sequence_length"] = feature_metadata["max_sequence_length"]
+        output_feature[DECODER]["vocab_size"] = feature_metadata["vocab_size"]
+        output_feature[DECODER]["max_sequence_length"] = feature_metadata["max_sequence_length"]
         if isinstance(output_feature[LOSS]["class_weights"], (list, tuple)):
-            if len(output_feature[LOSS]["class_weights"]) != output_feature["vocab_size"]:
+            if len(output_feature[LOSS]["class_weights"]) != output_feature[DECODER]["vocab_size"]:
                 raise ValueError(
                     "The length of class_weights ({}) is not compatible with "
                     "the number of classes ({}) for feature {}. "
@@ -384,7 +375,7 @@ def update_config_with_metadata(output_feature, feature_metadata, *args, **kwarg
                     "and their order and consider there needs to be a weight "
                     "for the <UNK> and <PAD> class too.".format(
                         len(output_feature[LOSS]["class_weights"]),
-                        output_feature["vocab_size"],
+                        output_feature[DECODER]["vocab_size"],
                         output_feature[COLUMN],
                     )
                 )
@@ -426,14 +417,14 @@ def update_config_with_metadata(output_feature, feature_metadata, *args, **kwarg
                         )
                     )
 
-                if all_rows_length != output_feature["vocab_size"]:
+                if all_rows_length != output_feature[DECODER]["vocab_size"]:
                     raise ValueError(
                         "The size of the class_similarities matrix of {} is "
                         "{}, different from the number of classes ({}). "
                         "Check the metadata JSON file to see the classes "
                         "and their order and "
                         "consider <UNK> and <PAD> class too.".format(
-                            output_feature[COLUMN], all_rows_length, output_feature["vocab_size"]
+                            output_feature[COLUMN], all_rows_length, output_feature[DECODER]["vocab_size"]
                         )
                     )
 
@@ -507,29 +498,25 @@ def last_idx2str(last_pred):
 
     @staticmethod
     def populate_defaults(output_feature):
-        set_default_value(
+        defaults = SequenceOutputFeatureConfig()
+        set_default_value(output_feature, LOSS, {})
+        set_default_values(output_feature[LOSS], defaults.loss)
+
+        if DECODER in output_feature and TYPE in output_feature[DECODER] and output_feature[DECODER][TYPE] == "tagger":
+            set_default_value(output_feature, "reduce_input", None)
+
+        set_default_values(
             output_feature,
-            LOSS,
             {
-                TYPE: "sequence_softmax_cross_entropy",
-                "class_weights": 1,
-                "robust_lambda": 0,
-                "confidence_penalty": 0,
-                "class_similarities_temperature": 0,
-                "weight": 1,
+                DECODER: {
+                    TYPE: defaults.decoder.type,
+                },
+                DEPENDENCIES: defaults.dependencies,
+                REDUCE_INPUT: defaults.reduce_input,
+                REDUCE_DEPENDENCIES: defaults.reduce_dependencies,
             },
         )
 
-        set_default_value(output_feature[LOSS], "unique", False)
-        set_default_value(output_feature, "decoder", "generator")
-
-        if output_feature["decoder"] == "tagger":
-            set_default_value(output_feature, "reduce_input", None)
-
-        set_default_value(output_feature, "dependencies", [])
-        set_default_value(output_feature, "reduce_input", SUM)
-        set_default_value(output_feature, "reduce_dependencies", SUM)
-
     @staticmethod
     def create_postproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
         return _SequencePostprocessing(metadata)
@@ -545,5 +532,7 @@ def flatten(self, df: DataFrame) -> DataFrame:
 
     def unflatten(self, df: DataFrame) -> DataFrame:
         probs_col = f"{self.feature_name}_{PROBABILITIES}"
-        df[probs_col] = df[probs_col].apply(lambda x: x.reshape(-1, self.num_classes), meta=(probs_col, "object"))
+        df[probs_col] = df[probs_col].apply(
+            lambda x: x.reshape(-1, self.decoder_config.num_classes), meta=(probs_col, "object")
+        )
         return df
diff --git a/ludwig/features/set_feature.py b/ludwig/features/set_feature.py
index f9af64b3627..912b36aa223 100644
--- a/ludwig/features/set_feature.py
+++ b/ludwig/features/set_feature.py
@@ -14,14 +14,16 @@
 # limitations under the License.
 # ==============================================================================
 import logging
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Union
 
 import numpy as np
 import torch
 
 from ludwig.constants import (
     COLUMN,
-    FILL_WITH_CONST,
+    DECODER,
+    DEPENDENCIES,
+    ENCODER,
     HIDDEN,
     JACCARD,
     LOGITS,
@@ -30,9 +32,10 @@
     PREDICTIONS,
     PROBABILITIES,
     PROC_COLUMN,
+    REDUCE_DEPENDENCIES,
+    REDUCE_INPUT,
     SET,
-    SIGMOID_CROSS_ENTROPY,
-    SUM,
+    THRESHOLD,
     TIED,
     TYPE,
 )
@@ -41,7 +44,7 @@
 from ludwig.schema.features.set_feature import SetInputFeatureConfig, SetOutputFeatureConfig
 from ludwig.schema.features.utils import register_input_feature, register_output_feature
 from ludwig.utils import output_feature_utils
-from ludwig.utils.misc_utils import get_from_registry, set_default_value
+from ludwig.utils.misc_utils import get_from_registry, set_default_value, set_default_values
 from ludwig.utils.strings_utils import create_vocabulary, tokenizer_registry, UNKNOWN_SYMBOL
 from ludwig.utils.tokenizers import TORCHSCRIPT_COMPATIBLE_TOKENIZERS
 from ludwig.utils.types import TorchscriptPreprocessingInput
@@ -159,13 +162,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "tokenizer": "space",
-            "most_common": 10000,
-            "lowercase": False,
-            "missing_value_strategy": FILL_WITH_CONST,
-            "fill_value": UNKNOWN_SYMBOL,
-        }
+        return SetInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -215,16 +212,14 @@ def add_feature_data(
 
 @register_input_feature(SET)
 class SetInputFeature(SetFeatureMixin, InputFeature):
-    encoder = "embed"
-    vocab = []
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
+    def __init__(self, input_feature_config: Union[SetInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs):
         assert isinstance(inputs, torch.Tensor)
@@ -240,15 +235,17 @@ def input_dtype(self):
 
     @property
     def input_shape(self) -> torch.Size:
-        return torch.Size([len(self.vocab)])
+        return torch.Size([len(self.encoder_config.vocab)])
 
     @staticmethod
     def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs):
-        input_feature["vocab"] = feature_metadata["idx2str"]
+        input_feature[ENCODER]["vocab"] = feature_metadata["idx2str"]
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
+        defaults = SetInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def get_schema_cls():
@@ -265,17 +262,18 @@ def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
 
 @register_output_feature(SET)
 class SetOutputFeature(SetFeatureMixin, OutputFeature):
-    decoder = "classifier"
-    loss = {TYPE: SIGMOID_CROSS_ENTROPY}
     metric_functions = {LOSS: None, JACCARD: None}
     default_validation_metric = JACCARD
-    num_classes = 0
-    threshold = 0.5
 
-    def __init__(self, feature, output_features: Dict[str, OutputFeature]):
-        super().__init__(feature, output_features)
-        self.overwrite_defaults(feature)
-        self.decoder_obj = self.initialize_decoder(feature)
+    def __init__(
+        self,
+        output_feature_config: Union[SetOutputFeatureConfig, Dict],
+        output_features: Dict[str, OutputFeature],
+        **kwargs,
+    ):
+        output_feature_config = self.load_config(output_feature_config)
+        super().__init__(output_feature_config, output_features, **kwargs)
+        self.decoder_obj = self.initialize_decoder()
         self._setup_loss()
         self._setup_metrics()
 
@@ -287,10 +285,10 @@ def loss_kwargs(self):
         return self.loss
 
     def metric_kwargs(self) -> Dict[str, Any]:
-        return {"threshold": self.threshold}
+        return {"threshold": self.decoder_config.threshold}
 
     def create_predict_module(self) -> PredictModule:
-        return _SetPredict(self.threshold)
+        return _SetPredict(self.decoder_config.threshold)
 
     def get_prediction_set(self):
         return {PREDICTIONS, PROBABILITIES, LOGITS}
@@ -305,13 +303,13 @@ def input_shape(self) -> torch.Size:
 
     @property
     def output_shape(self) -> torch.Size:
-        return torch.Size([self.num_classes])
+        return torch.Size([self.decoder_config.num_classes])
 
     @staticmethod
     def update_config_with_metadata(output_feature, feature_metadata, *args, **kwargs):
-        output_feature["num_classes"] = feature_metadata["vocab_size"]
+        output_feature[DECODER]["num_classes"] = feature_metadata["vocab_size"]
         if isinstance(output_feature[LOSS]["class_weights"], (list, tuple)):
-            if len(output_feature[LOSS]["class_weights"]) != output_feature["num_classes"]:
+            if len(output_feature[LOSS]["class_weights"]) != output_feature[DECODER]["num_classes"]:
                 raise ValueError(
                     "The length of class_weights ({}) is not compatible with "
                     "the number of classes ({}) for feature {}. "
@@ -361,7 +359,7 @@ def idx2str(pred_set):
 
         probabilities_col = f"{self.feature_name}_{PROBABILITIES}"
         if probabilities_col in result:
-            threshold = self.threshold
+            threshold = self.decoder_config.threshold
 
             def get_prob(prob_set):
                 # Cast to float32 because empty np.array objects are np.float64, causing mismatch errors during saving.
@@ -377,14 +375,22 @@ def create_postproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
 
     @staticmethod
     def populate_defaults(output_feature):
-        set_default_value(output_feature, LOSS, {TYPE: SIGMOID_CROSS_ENTROPY, "weight": 1})
-        set_default_value(output_feature[LOSS], "weight", 1)
-        set_default_value(output_feature[LOSS], "class_weights", None)
-
-        set_default_value(output_feature, "threshold", 0.5)
-        set_default_value(output_feature, "dependencies", [])
-        set_default_value(output_feature, "reduce_input", SUM)
-        set_default_value(output_feature, "reduce_dependencies", SUM)
+        defaults = SetOutputFeatureConfig()
+        set_default_value(output_feature, LOSS, {})
+        set_default_values(output_feature[LOSS], defaults.loss)
+
+        set_default_values(
+            output_feature,
+            {
+                DECODER: {
+                    TYPE: defaults.decoder.type,
+                    THRESHOLD: defaults.decoder.threshold,
+                },
+                DEPENDENCIES: defaults.dependencies,
+                REDUCE_INPUT: defaults.reduce_input,
+                REDUCE_DEPENDENCIES: defaults.reduce_dependencies,
+            },
+        )
 
     @staticmethod
     def get_schema_cls():
diff --git a/ludwig/features/text_feature.py b/ludwig/features/text_feature.py
index b57893bedee..5a93b5ed0fa 100644
--- a/ludwig/features/text_feature.py
+++ b/ludwig/features/text_feature.py
@@ -15,14 +15,15 @@
 # ==============================================================================
 import logging
 from functools import partial
-from typing import Any, Dict
+from typing import Any, Dict, Union
 
 import torch
 
 from ludwig.constants import (
     COLUMN,
+    DECODER,
     EDIT_DISTANCE,
-    FILL_WITH_CONST,
+    ENCODER,
     LAST_ACCURACY,
     LAST_PREDICTIONS,
     LENGTHS,
@@ -50,14 +51,8 @@
 from ludwig.schema.features.text_feature import TextInputFeatureConfig, TextOutputFeatureConfig
 from ludwig.schema.features.utils import register_input_feature, register_output_feature
 from ludwig.utils.math_utils import softmax
-from ludwig.utils.misc_utils import set_default_values
-from ludwig.utils.strings_utils import (
-    build_sequence_matrix,
-    create_vocabulary,
-    PADDING_SYMBOL,
-    SpecialSymbol,
-    UNKNOWN_SYMBOL,
-)
+from ludwig.utils.misc_utils import set_default_value, set_default_values
+from ludwig.utils.strings_utils import build_sequence_matrix, create_vocabulary, SpecialSymbol, UNKNOWN_SYMBOL
 from ludwig.utils.types import DataFrame
 
 logger = logging.getLogger(__name__)
@@ -70,19 +65,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "tokenizer": "space_punct",
-            "pretrained_model_name_or_path": None,
-            "vocab_file": None,
-            "max_sequence_length": 256,
-            "most_common": 20000,
-            "padding_symbol": PADDING_SYMBOL,
-            "unknown_symbol": UNKNOWN_SYMBOL,
-            "padding": "right",
-            "lowercase": True,
-            "missing_value_strategy": FILL_WITH_CONST,
-            "fill_value": UNKNOWN_SYMBOL,
-        }
+        return TextInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -200,12 +183,11 @@ def add_feature_data(
 
 @register_input_feature(TEXT)
 class TextInputFeature(TextFeatureMixin, SequenceInputFeature):
-    encoder = "parallel_cnn"
-    max_sequence_length = None
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature, encoder_obj=encoder_obj)
-        self._input_shape = [feature["max_sequence_length"]]
+    def __init__(self, input_feature_config: Union[TextInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, encoder_obj=encoder_obj, **kwargs)
+        self.encoder_config = input_feature_config.encoder
+        # self._input_shape = [input_feature_config.encoder.max_sequence_length]
 
     def forward(self, inputs, mask=None):
         assert isinstance(inputs, torch.Tensor)
@@ -232,20 +214,22 @@ def input_dtype(self):
 
     @property
     def input_shape(self):
-        return torch.Size(self._input_shape)
+        return torch.Size([self.encoder_config.max_sequence_length])
 
     @staticmethod
     def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs):
-        input_feature["vocab"] = feature_metadata["idx2str"]
-        input_feature["max_sequence_length"] = feature_metadata["max_sequence_length"]
-        input_feature["pad_idx"] = feature_metadata["pad_idx"]
-        input_feature["num_tokens"] = len(feature_metadata["idx2str"])
+        input_feature[ENCODER]["vocab"] = feature_metadata["idx2str"]
+        input_feature[ENCODER]["max_sequence_length"] = feature_metadata["max_sequence_length"]
+        input_feature[ENCODER]["pad_idx"] = feature_metadata["pad_idx"]
+        input_feature[ENCODER]["num_tokens"] = len(feature_metadata["idx2str"])
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_values(input_feature, {TIED: None, "encoder": "parallel_cnn"})
+        defaults = TextInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
-        encoder_class = get_encoder_cls(input_feature["type"], input_feature["encoder"])
+        encoder_class = get_encoder_cls(input_feature[TYPE], input_feature[ENCODER][TYPE])
 
         if hasattr(encoder_class, "default_params"):
             set_default_values(input_feature, encoder_class.default_params)
@@ -265,14 +249,17 @@ def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
 
 @register_output_feature(TEXT)
 class TextOutputFeature(TextFeatureMixin, SequenceOutputFeature):
-    loss = {TYPE: "sequence_softmax_cross_entropy"}
     metric_functions = {LOSS: None, TOKEN_ACCURACY: None, LAST_ACCURACY: None, PERPLEXITY: None, EDIT_DISTANCE: None}
     default_validation_metric = LOSS
-    max_sequence_length = 0
-    vocab_size = 0
 
-    def __init__(self, feature, output_features: Dict[str, OutputFeature]):
-        super().__init__(feature, output_features)
+    def __init__(
+        self,
+        output_feature_config: Union[TextInputFeatureConfig, Dict],
+        output_features: Dict[str, OutputFeature],
+        **kwargs,
+    ):
+        output_feature_config = self.load_config(output_feature_config)
+        super().__init__(output_feature_config, output_features, **kwargs)
 
     @classmethod
     def get_output_dtype(cls):
@@ -280,20 +267,20 @@ def get_output_dtype(cls):
 
     @property
     def output_shape(self) -> torch.Size:
-        return torch.Size([self.max_sequence_length])
+        return torch.Size([self.decoder_config.max_sequence_length])
 
     @staticmethod
     def update_config_with_metadata(output_feature, feature_metadata, *args, **kwargs):
-        output_feature["vocab_size"] = feature_metadata["vocab_size"]
-        output_feature["max_sequence_length"] = feature_metadata["max_sequence_length"]
+        output_feature[DECODER]["vocab_size"] = feature_metadata["vocab_size"]
+        output_feature[DECODER]["max_sequence_length"] = feature_metadata["max_sequence_length"]
         if isinstance(output_feature[LOSS]["class_weights"], (list, tuple)):
             # [0, 0] for UNK and PAD
             output_feature[LOSS]["class_weights"] = [0, 0] + output_feature[LOSS]["class_weights"]
-            if len(output_feature[LOSS]["class_weights"]) != output_feature["vocab_size"]:
+            if len(output_feature[LOSS]["class_weights"]) != output_feature[DECODER]["vocab_size"]:
                 raise ValueError(
                     "The length of class_weights ({}) is not compatible with "
                     "the number of classes ({})".format(
-                        len(output_feature[LOSS]["class_weights"]), output_feature["vocab_size"]
+                        len(output_feature[LOSS]["class_weights"]), output_feature[DECODER]["vocab_size"]
                     )
                 )
 
@@ -386,6 +373,6 @@ def flatten(self, df: DataFrame) -> DataFrame:
     def unflatten(self, df: DataFrame) -> DataFrame:
         probs_col = f"{self.feature_name}_{PROBABILITIES}"
         df[probs_col] = df[probs_col].apply(
-            lambda x: x.reshape(-1, self.max_sequence_length), meta=(probs_col, "object")
+            lambda x: x.reshape(-1, self.decoder_config.max_sequence_length), meta=(probs_col, "object")
         )
         return df
diff --git a/ludwig/features/timeseries_feature.py b/ludwig/features/timeseries_feature.py
index 342517cc1af..ba027723ddb 100644
--- a/ludwig/features/timeseries_feature.py
+++ b/ludwig/features/timeseries_feature.py
@@ -14,17 +14,17 @@
 # limitations under the License.
 # ==============================================================================
 import logging
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Union
 
 import numpy as np
 import torch
 
-from ludwig.constants import COLUMN, FILL_WITH_CONST, NAME, PROC_COLUMN, TIED, TIMESERIES
+from ludwig.constants import COLUMN, ENCODER, NAME, PROC_COLUMN, TIED, TIMESERIES, TYPE
 from ludwig.features.base_feature import BaseFeatureMixin
 from ludwig.features.sequence_feature import SequenceInputFeature
 from ludwig.schema.features.timeseries_feature import TimeseriesInputFeatureConfig
 from ludwig.schema.features.utils import register_input_feature
-from ludwig.utils.misc_utils import get_from_registry, set_default_values
+from ludwig.utils.misc_utils import get_from_registry, set_default_value, set_default_values
 from ludwig.utils.strings_utils import tokenizer_registry
 from ludwig.utils.tokenizers import TORCHSCRIPT_COMPATIBLE_TOKENIZERS
 from ludwig.utils.types import TorchscriptPreprocessingInput
@@ -109,14 +109,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "timeseries_length_limit": 256,
-            "padding_value": 0,
-            "padding": "right",
-            "tokenizer": "space",
-            "missing_value_strategy": FILL_WITH_CONST,
-            "fill_value": "",
-        }
+        return TimeseriesInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -183,16 +176,15 @@ def add_feature_data(
 
 @register_input_feature(TIMESERIES)
 class TimeseriesInputFeature(TimeseriesFeatureMixin, SequenceInputFeature):
-    encoder = "parallel_cnn"
-    max_sequence_length = None
-
-    def __init__(self, feature, encoder_obj=None):
+    def __init__(self, input_feature_config: Union[TimeseriesInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
         # add required sequence encoder parameters for time series
-        feature["embedding_size"] = 1
-        feature["should_embed"] = False
+        self.encoder_config = input_feature_config.encoder
+        self.encoder_config.embedding_size = 1
+        self.encoder_config.should_embed = False
 
-        # initialize encoder for time series
-        super().__init__(feature, encoder_obj=encoder_obj)
+        # SequenceInputFeauture's constructor initializes the encoder.
+        super().__init__(input_feature_config, encoder_obj=encoder_obj, **kwargs)
 
     def forward(self, inputs, mask=None):
         assert isinstance(inputs, torch.Tensor)
@@ -206,7 +198,7 @@ def forward(self, inputs, mask=None):
 
     @property
     def input_shape(self) -> torch.Size:
-        return torch.Size([self.max_sequence_length])
+        return torch.Size([self.encoder_config.max_sequence_length])
 
     @property
     def input_dtype(self):
@@ -214,17 +206,13 @@ def input_dtype(self):
 
     @staticmethod
     def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs):
-        input_feature["max_sequence_length"] = feature_metadata["max_timeseries_length"]
+        input_feature[ENCODER]["max_sequence_length"] = feature_metadata["max_timeseries_length"]
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_values(
-            input_feature,
-            {
-                TIED: None,
-                "encoder": "parallel_cnn",
-            },
-        )
+        defaults = TimeseriesInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
 
     @staticmethod
     def get_schema_cls():
@@ -248,9 +236,7 @@ def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
 #             'class_similarities_temperature': 0
 #         }
 #         self.num_classes = 0
-#
-#         self.overwrite_defaults(feature)
-#
+##
 #         self.decoder_obj = self.get_sequence_decoder(feature)
 #
 #     def _get_output_placeholder(self):
diff --git a/ludwig/features/vector_feature.py b/ludwig/features/vector_feature.py
index b90473f7f3e..8d9a1670f1c 100644
--- a/ludwig/features/vector_feature.py
+++ b/ludwig/features/vector_feature.py
@@ -14,15 +14,17 @@
 # limitations under the License.
 # ==============================================================================
 import logging
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Union
 
 import numpy as np
 import torch
 
 from ludwig.constants import (
     COLUMN,
+    DECODER,
+    DEPENDENCIES,
+    ENCODER,
     ERROR,
-    FILL_WITH_CONST,
     HIDDEN,
     LOGITS,
     LOSS,
@@ -30,8 +32,11 @@
     MEAN_SQUARED_ERROR,
     NAME,
     PREDICTIONS,
+    PREPROCESSING,
     PROC_COLUMN,
     R2,
+    REDUCE_DEPENDENCIES,
+    REDUCE_INPUT,
     TIED,
     TYPE,
     VECTOR,
@@ -40,8 +45,7 @@
 from ludwig.schema.features.utils import register_input_feature, register_output_feature
 from ludwig.schema.features.vector_feature import VectorInputFeatureConfig, VectorOutputFeatureConfig
 from ludwig.utils import output_feature_utils
-from ludwig.utils.misc_utils import set_default_value
-from ludwig.utils.torch_utils import LudwigModule
+from ludwig.utils.misc_utils import set_default_value, set_default_values
 from ludwig.utils.types import TorchscriptPreprocessingInput
 
 logger = logging.getLogger(__name__)
@@ -94,10 +98,7 @@ def type():
 
     @staticmethod
     def preprocessing_defaults():
-        return {
-            "missing_value_strategy": FILL_WITH_CONST,
-            "fill_value": "",
-        }
+        return VectorInputFeatureConfig().preprocessing.__dict__
 
     @staticmethod
     def cast_column(column, backend):
@@ -147,17 +148,15 @@ def add_feature_data(
 
 @register_input_feature(VECTOR)
 class VectorInputFeature(VectorFeatureMixin, InputFeature):
-    encoder = "dense"
-    vector_size = 0
-
-    def __init__(self, feature: Dict[str, Any], encoder_obj: Optional[LudwigModule] = None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
-        feature["input_size"] = feature["vector_size"]
+    def __init__(self, input_feature_config: Union[VectorInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
+        # self.encoder_config.input_size = self.encoder_config.vector_size
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
 
     def forward(self, inputs: torch.Tensor) -> torch.Tensor:
         assert isinstance(inputs, torch.Tensor)
@@ -170,7 +169,7 @@ def forward(self, inputs: torch.Tensor) -> torch.Tensor:
 
     @property
     def input_shape(self) -> torch.Size:
-        return torch.Size([self.vector_size])
+        return torch.Size([self.encoder_config.input_size])
 
     @property
     def output_shape(self) -> torch.Size:
@@ -178,13 +177,14 @@ def output_shape(self) -> torch.Size:
 
     @staticmethod
     def update_config_with_metadata(input_feature, feature_metadata, *args, **kwargs):
-        for key in ["vector_size"]:
-            input_feature[key] = feature_metadata[key]
+        input_feature[ENCODER]["input_size"] = feature_metadata["vector_size"]
 
     @staticmethod
     def populate_defaults(input_feature):
-        set_default_value(input_feature, TIED, None)
-        set_default_value(input_feature, "preprocessing", {})
+        defaults = VectorInputFeatureConfig()
+        set_default_value(input_feature, TIED, defaults.tied)
+        set_default_values(input_feature, {ENCODER: {TYPE: defaults.encoder.type}})
+        set_default_value(input_feature, PREPROCESSING, {})
 
     @staticmethod
     def create_preproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
@@ -197,18 +197,19 @@ def get_schema_cls():
 
 @register_output_feature(VECTOR)
 class VectorOutputFeature(VectorFeatureMixin, OutputFeature):
-    decoder = "projector"
-    loss = {TYPE: MEAN_SQUARED_ERROR}
     metric_functions = {LOSS: None, ERROR: None, MEAN_SQUARED_ERROR: None, MEAN_ABSOLUTE_ERROR: None, R2: None}
     default_validation_metric = MEAN_SQUARED_ERROR
-    vector_size = 0
-
-    def __init__(self, feature, output_features: Dict[str, OutputFeature]):
-        super().__init__(feature, output_features)
-        self.overwrite_defaults(feature)
-        self._input_shape = feature["input_size"]
-        feature["output_size"] = feature["vector_size"]
-        self.decoder_obj = self.initialize_decoder(feature)
+
+    def __init__(
+        self,
+        output_feature_config: Union[VectorOutputFeatureConfig, Dict],
+        output_features: Dict[str, OutputFeature],
+        **kwargs,
+    ):
+        output_feature_config = self.load_config(output_feature_config)
+        super().__init__(output_feature_config, output_features, **kwargs)
+        self.decoder_config.output_size = self.decoder_config.vector_size
+        self.decoder_obj = self.initialize_decoder()
         self._setup_loss()
         self._setup_metrics()
 
@@ -234,15 +235,15 @@ def get_output_dtype(cls):
 
     @property
     def output_shape(self) -> torch.Size:
-        return torch.Size([self.vector_size])
+        return torch.Size([self.decoder_config.vector_size])
 
     @property
     def input_shape(self) -> torch.Size:
-        return torch.Size([self._input_shape])
+        return torch.Size([self.decoder_config.input_size])
 
     @staticmethod
     def update_config_with_metadata(output_feature, feature_metadata, *args, **kwargs):
-        output_feature["vector_size"] = feature_metadata["vector_size"]
+        output_feature[DECODER]["vector_size"] = feature_metadata["vector_size"]
 
     @staticmethod
     def calculate_overall_stats(predictions, targets, train_set_metadata):
@@ -261,13 +262,23 @@ def postprocess_predictions(
 
     @staticmethod
     def populate_defaults(output_feature):
+        defaults = VectorOutputFeatureConfig()
+
+        # If Loss is not defined, set an empty dictionary
         set_default_value(output_feature, LOSS, {})
-        set_default_value(output_feature[LOSS], TYPE, MEAN_SQUARED_ERROR)
-        set_default_value(output_feature[LOSS], "weight", 1)
-        set_default_value(output_feature, "reduce_input", None)
-        set_default_value(output_feature, "reduce_dependencies", None)
-        set_default_value(output_feature, "decoder", "projector")
-        set_default_value(output_feature, "dependencies", [])
+        set_default_values(output_feature[LOSS], defaults.loss)
+
+        set_default_values(
+            output_feature,
+            {
+                DECODER: {
+                    TYPE: defaults.decoder.type,
+                },
+                DEPENDENCIES: defaults.dependencies,
+                REDUCE_INPUT: defaults.reduce_input,
+                REDUCE_DEPENDENCIES: defaults.reduce_dependencies,
+            },
+        )
 
     @staticmethod
     def create_postproc_module(metadata: Dict[str, Any]) -> torch.nn.Module:
diff --git a/ludwig/hyperopt/execution.py b/ludwig/hyperopt/execution.py
index 3ee78ff56f0..781b2a8340d 100644
--- a/ludwig/hyperopt/execution.py
+++ b/ludwig/hyperopt/execution.py
@@ -30,7 +30,9 @@
 from ludwig.constants import (
     COLUMN,
     COMBINER,
+    DECODER,
     DEFAULTS,
+    ENCODER,
     INPUT_FEATURES,
     MAXIMIZE,
     OUTPUT_FEATURES,
@@ -964,6 +966,17 @@ def update_features_with_shared_params(
         return
 
     sampled_default_shared_params = trial_parameters_dict.get(DEFAULTS).get(feature_type)
+    shared_params_copy = copy.deepcopy(sampled_default_shared_params)
+
+    # Remove encoder/decoder from output/input features
+    if config_feature_group == INPUT_FEATURES:
+        if DECODER in sampled_default_shared_params:
+            del shared_params_copy[DECODER]
+    else:
+        if ENCODER in sampled_default_shared_params:
+            del shared_params_copy[ENCODER]
+    sampled_default_shared_params = shared_params_copy
+
     set_values(sampled_default_shared_params, section_dict)
 
 
diff --git a/ludwig/hyperopt/run.py b/ludwig/hyperopt/run.py
index aaea14865ec..0b70bfdae96 100644
--- a/ludwig/hyperopt/run.py
+++ b/ludwig/hyperopt/run.py
@@ -28,10 +28,9 @@
     VALIDATION,
 )
 from ludwig.data.split import get_splitter
-from ludwig.features.feature_registries import output_type_registry
+from ludwig.features.feature_registries import input_type_registry, output_type_registry
 from ludwig.hyperopt.results import HyperoptResults
 from ludwig.hyperopt.utils import print_hyperopt_results, save_hyperopt_stats, should_tune_preprocessing
-from ludwig.utils.config_utils import get_default_encoder_or_decoder
 from ludwig.utils.defaults import default_random_seed, merge_with_defaults
 from ludwig.utils.fs_utils import makedirs, open_file
 from ludwig.utils.misc_utils import get_class_attributes, get_from_registry, set_default_value, set_default_values
@@ -274,8 +273,8 @@ def hyperopt(
         feature_class = get_from_registry(output_feature_type, output_type_registry)
         if metric not in feature_class.metric_functions:
             # todo v0.4: allow users to specify also metrics from the overall
-            #  and per class metrics from the trainign stats and in general
-            #  and potprocessed metric
+            #  and per class metrics from the training stats and in general
+            #  and post-processed metric
             raise ValueError(
                 'The specified metric for hyperopt "{}" is not a valid metric '
                 'for the specified output feature "{}" of type "{}". '
@@ -440,12 +439,14 @@ def get_features_eligible_for_shared_params(
         if TYPE not in feature:
             raise ValueError("Ludwig expects feature types to be defined for each feature within the config.")
         if config_feature_type == INPUT_FEATURES:
-            default_encoder = get_default_encoder_or_decoder(feature, INPUT_FEATURES)
-            if not feature.get(ENCODER, 0) or feature.get(ENCODER) == default_encoder:
+            feature_schema = get_from_registry(feature.get(TYPE), input_type_registry).get_schema_cls()
+            default_encoder = feature_schema().encoder.type
+            if not feature[ENCODER].get(TYPE, 0) or feature[ENCODER].get(TYPE) == default_encoder:
                 features_eligible_for_shared_params[feature[TYPE]].add(feature[NAME])
         else:
-            default_decoder = get_default_encoder_or_decoder(feature, OUTPUT_FEATURES)
-            if not feature.get(DECODER, 0) or feature.get(DECODER) == default_decoder:
+            feature_schema = get_from_registry(feature.get(TYPE), output_type_registry).get_schema_cls()
+            default_decoder = feature_schema().decoder.type
+            if not feature[DECODER].get(TYPE, 0) or feature[DECODER].get(TYPE) == default_decoder:
                 features_eligible_for_shared_params[feature[TYPE]].add(feature[NAME])
 
     return features_eligible_for_shared_params
diff --git a/ludwig/models/base.py b/ludwig/models/base.py
index fbd29b965f7..ce6df66e8cd 100644
--- a/ludwig/models/base.py
+++ b/ludwig/models/base.py
@@ -7,7 +7,7 @@
 import torch
 
 from ludwig.combiners.combiners import Combiner
-from ludwig.constants import COMBINED, LOSS, NAME, TIED, TYPE
+from ludwig.constants import COMBINED, DECODER, LOSS, NAME, TIED, TYPE
 from ludwig.features.base_feature import InputFeature, OutputFeature
 from ludwig.features.feature_registries import input_type_registry, output_type_registry
 from ludwig.utils.algorithms_utils import topological_sort_feature_dependencies
@@ -16,6 +16,8 @@
 from ludwig.utils.torch_utils import DEVICE, LudwigModule, reg_loss
 from ludwig.utils.types import TorchDevice
 
+logger = logging.getLogger(__name__)
+
 
 class BaseModel(LudwigModule, metaclass=ABCMeta):
     """Base model for use in LudwigModule.
@@ -51,10 +53,10 @@ def build_inputs(cls, input_features_def: List[Dict[str, Any]]) -> Dict[str, Inp
 
     @staticmethod
     def build_single_input(
-        input_feature_def: Dict[str, Any], other_input_features: Dict[str, InputFeature]
+        input_feature_def: Dict[str, Any], other_input_features: Optional[Dict[str, InputFeature]]
     ) -> InputFeature:
         """Builds a single input feature from the input feature definition."""
-        logging.debug(f"Input {input_feature_def[TYPE]} feature {input_feature_def[NAME]}")
+        logger.debug(f"Input {input_feature_def[TYPE]} feature {input_feature_def[NAME]}")
 
         encoder_obj = None
         if input_feature_def.get(TIED, None) is not None:
@@ -63,8 +65,7 @@ def build_single_input(
                 encoder_obj = other_input_features[tied_input_feature_name].encoder_obj
 
         input_feature_class = get_from_registry(input_feature_def[TYPE], input_type_registry)
-        input_feature_obj = input_feature_class(input_feature_def, encoder_obj)
-
+        input_feature_obj = input_feature_class(input_feature_def, encoder_obj=encoder_obj)
         return input_feature_obj
 
     @classmethod
@@ -76,7 +77,7 @@ def build_outputs(cls, output_features_def: List[Dict[str, Any]], combiner: Comb
         for output_feature_def in output_features_def:
             # TODO(Justin): Check that the semantics of input_size align with what the combiner's output shape returns
             # for seq2seq.
-            output_feature_def["input_size"] = combiner.output_shape[-1]
+            output_feature_def[DECODER]["input_size"] = combiner.output_shape[-1]
             output_feature = cls.build_single_output(output_feature_def, output_features)
             output_features[output_feature_def[NAME]] = output_feature
 
@@ -84,14 +85,12 @@ def build_outputs(cls, output_features_def: List[Dict[str, Any]], combiner: Comb
 
     @staticmethod
     def build_single_output(
-        output_feature_def: Dict[str, Any], output_features: Dict[str, OutputFeature]
+        output_feature_def: Dict[str, Any], output_features: Optional[Dict[str, OutputFeature]]
     ) -> OutputFeature:
         """Builds a single output feature from the output feature definition."""
-        logging.debug(f"Output {output_feature_def[TYPE]} feature {output_feature_def[NAME]}")
-
+        logger.debug(f"Output {output_feature_def[TYPE]} feature {output_feature_def[NAME]}")
         output_feature_class = get_from_registry(output_feature_def[TYPE], output_type_registry)
-        output_feature_obj = output_feature_class(output_feature_def, output_features)
-
+        output_feature_obj = output_feature_class(output_feature_def, output_features=output_features)
         return output_feature_obj
 
     def get_model_inputs(self):
diff --git a/ludwig/models/ecd.py b/ludwig/models/ecd.py
index a8b3dec2997..d9375a59eec 100644
--- a/ludwig/models/ecd.py
+++ b/ludwig/models/ecd.py
@@ -17,6 +17,8 @@
 from ludwig.utils.data_utils import clear_data_cache
 from ludwig.utils.torch_utils import get_torch_device
 
+logger = logging.getLogger(__name__)
+
 
 class ECD(BaseModel):
     @staticmethod
@@ -49,7 +51,7 @@ def __init__(
             )
 
         # ================ Combiner ================
-        logging.debug(f"Combiner {self._combiner_def[TYPE]}")
+        logger.debug(f"Combiner {self._combiner_def[TYPE]}")
         combiner_class = get_combiner_class(self._combiner_def[TYPE])
         config, kwargs = load_config_with_kwargs(
             combiner_class.get_schema_cls(),
diff --git a/ludwig/models/gbm.py b/ludwig/models/gbm.py
index 4f4e0db6adf..021ae4d2a0f 100644
--- a/ludwig/models/gbm.py
+++ b/ludwig/models/gbm.py
@@ -8,7 +8,7 @@
 import torchmetrics
 from hummingbird.ml import convert
 
-from ludwig.constants import BINARY, CATEGORY, LOGITS, MODEL_GBM, NAME, NUMBER
+from ludwig.constants import BINARY, CATEGORY, DECODER, LOGITS, MODEL_GBM, NAME, NUMBER
 from ludwig.features.base_feature import OutputFeature
 from ludwig.features.feature_utils import LudwigFeatureDict
 from ludwig.globals import MODEL_WEIGHTS_FILE_NAME
@@ -64,7 +64,7 @@ def build_outputs(cls, output_features_def: List[Dict[str, Any]], input_size: in
         output_feature_def = output_features_def[0]
         output_features = {}
 
-        output_feature_def["input_size"] = input_size
+        output_feature_def[DECODER]["input_size"] = input_size
         output_feature = cls.build_single_output(output_feature_def, output_features)
         output_features[output_feature_def[NAME]] = output_feature
 
@@ -85,7 +85,9 @@ def compile(self):
         gbm_sklearn.fitted_ = True
         gbm_sklearn._n_features = len(self.input_features)
         if isinstance(gbm_sklearn, lgb.LGBMClassifier):
-            gbm_sklearn._n_classes = output_feature.num_classes if output_feature.type() == CATEGORY else 2
+            gbm_sklearn._n_classes = (
+                output_feature.decoder_config.num_classes if output_feature.type() == CATEGORY else 2
+            )
 
         hb_model = convert(gbm_sklearn, "torch", extra_config={"tree_implementation": "gemm"})
 
diff --git a/ludwig/modules/fully_connected_modules.py b/ludwig/modules/fully_connected_modules.py
index 058fe84bad4..bbf6eaaf99b 100644
--- a/ludwig/modules/fully_connected_modules.py
+++ b/ludwig/modules/fully_connected_modules.py
@@ -131,7 +131,7 @@ def __init__(
         default_norm: Optional[str] = None,
         default_norm_params: Optional[Dict] = None,
         default_activation: str = "relu",
-        default_dropout: int = 0,
+        default_dropout: float = 0,
         residual: bool = False,
         **kwargs,
     ):
diff --git a/ludwig/schema/combiners/__init__.py b/ludwig/schema/combiners/__init__.py
index 9f084ae9233..e69de29bb2d 100644
--- a/ludwig/schema/combiners/__init__.py
+++ b/ludwig/schema/combiners/__init__.py
@@ -1,9 +0,0 @@
-# Pull configs here for convenient imports in other modules:
-from ludwig.schema.combiners.comparator import ComparatorCombinerConfig  # noqa: F401
-from ludwig.schema.combiners.concat import ConcatCombinerConfig  # noqa: F401
-from ludwig.schema.combiners.project_aggregate import ProjectAggregateCombinerConfig  # noqa: F401
-from ludwig.schema.combiners.sequence import SequenceCombinerConfig  # noqa: F401
-from ludwig.schema.combiners.sequence_concat import SequenceConcatCombinerConfig  # noqa: F401
-from ludwig.schema.combiners.tab_transformer import TabTransformerCombinerConfig  # noqa: F401
-from ludwig.schema.combiners.tabnet import TabNetCombinerConfig  # noqa: F401
-from ludwig.schema.combiners.transformer import TransformerCombinerConfig  # noqa: F401
diff --git a/ludwig/schema/decoders/__init__.py b/ludwig/schema/decoders/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/ludwig/schema/decoders/base.py b/ludwig/schema/decoders/base.py
new file mode 100644
index 00000000000..ea7312e5868
--- /dev/null
+++ b/ludwig/schema/decoders/base.py
@@ -0,0 +1,218 @@
+from abc import ABC
+from typing import List, Tuple, Union
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+
+
+@dataclass
+class BaseDecoderConfig(schema_utils.BaseMarshmallowConfig, ABC):
+    """Base class for decoders.
+
+    Not meant to be used directly.
+    """
+
+    type: str
+    "Name corresponding to a decoder."
+
+    input_size: int = None
+
+    num_classes: int = None
+
+    fc_layers: List[dict] = None
+
+    num_fc_layers: int = 0
+
+    output_size: int = 256
+
+    use_bias: bool = True
+
+    weights_initializer: str = "xavier_uniform"
+
+    bias_initializer: str = "zeros"
+
+    norm: str = None
+
+    norm_params: dict = None
+
+    activation: str = "relu"
+
+    dropout: float = 0
+
+
+@dataclass
+class PassthroughDecoderConfig(BaseDecoderConfig):
+    """PassthroughDecoderConfig is a dataclass that configures the parameters used for a passthrough decoder."""
+
+    type: str = "passthrough"
+
+
+@dataclass
+class RegressorConfig(BaseDecoderConfig):
+    """RegressorConfig is a dataclass that configures the parameters used for a regressor decoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["regressor"],
+        default="regressor",
+        allow_none=False,
+        description="Type of decoder.",
+    )
+
+    input_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Size of the input to the decoder.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer for the weight matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer for the bias vector.",
+    )
+
+    threshold: float = schema_utils.FloatRange(
+        default=0.5,
+        min=0,
+        max=1,
+        description="Threshold for the output of the decoder.",
+    )
+
+    clip: Union[List[int], Tuple[int]] = schema_utils.FloatRangeTupleDataclassField(
+        n=2,
+        default=None,
+        allow_none=True,
+        min=0,
+        max=999999999,
+        description="Clip the output of the decoder to be within the given range.",
+    )
+
+
+@dataclass
+class ProjectorConfig(BaseDecoderConfig):
+    """ProjectorConfig is a dataclass that configures the parameters used for a projector decoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["projector"],
+        default="projector",
+        allow_none=False,
+        description="Type of decoder.",
+    )
+
+    input_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Size of the input to the decoder.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Size of the output of the decoder.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer for the weight matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer for the bias vector.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        default=None,
+        description=" Indicates the activation function applied to the output.",
+    )
+
+    vector_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Size of the output of the decoder.",
+    )
+
+    clip: Union[List[int], Tuple[int]] = schema_utils.FloatRangeTupleDataclassField(
+        n=2,
+        default=None,
+        allow_none=True,
+        min=0,
+        max=999999999,
+        description="Clip the output of the decoder to be within the given range.",
+    )
+
+
+@dataclass
+class ClassifierConfig(BaseDecoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["classifier"],
+        default="classifier",
+        allow_none=False,
+        description="Type of decoder.",
+    )
+
+    input_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Size of the input to the decoder.",
+    )
+
+    num_classes: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of classes to predict.",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The default output_size that will be used for each layer.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer for the weight matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer for the bias vector.",
+    )
+
+    threshold: float = schema_utils.FloatRange(
+        default=0.5,
+        min=0,
+        max=1,
+        description="The threshold above (greater or equal) which the predicted output of the sigmoid will be mapped "
+        "to 1.",
+    )
diff --git a/ludwig/schema/decoders/sequence_decoders.py b/ludwig/schema/decoders/sequence_decoders.py
new file mode 100644
index 00000000000..30274a5fb24
--- /dev/null
+++ b/ludwig/schema/decoders/sequence_decoders.py
@@ -0,0 +1,94 @@
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.decoders.base import BaseDecoderConfig
+
+
+@dataclass
+class SequenceGeneratorDecoderConfig(BaseDecoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["generator"],
+        default="generator",
+        allow_none=False,
+        description="Type of decoder.",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Size of the vocabulary.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the sequences.",
+    )
+
+    cell_type: str = schema_utils.StringOptions(
+        ["rnn", "lstm", "gru"],
+        default="gru",
+        description="Type of recurrent cell to use.",
+    )
+
+    input_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Size of the input to the decoder.",
+    )
+
+    reduce_input: str = schema_utils.StringOptions(
+        ["sum", "mean", "avg", "max", "concat", "last"],
+        default="sum",
+        description="How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first "
+        "dimension (second if you count the batch dimension)",
+    )
+
+    num_layers: int = schema_utils.PositiveInteger(
+        default=1,
+        description="The number of stacked recurrent layers.",
+    )
+
+
+@dataclass
+class SequenceTaggerDecoderConfig(BaseDecoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["tagger"],
+        default="tagger",
+        allow_none=False,
+        description="Type of decoder.",
+    )
+
+    input_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Size of the input to the decoder.",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Size of the vocabulary.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the sequences.",
+    )
+
+    use_attention: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to apply a multi-head self attention layer before prediction.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    attention_embedding_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The embedding size of the multi-head self attention layer.",
+    )
+
+    attention_num_heads: int = schema_utils.PositiveInteger(
+        default=8,
+        description="The number of attention heads in the multi-head self attention layer.",
+    )
diff --git a/ludwig/schema/decoders/utils.py b/ludwig/schema/decoders/utils.py
new file mode 100644
index 00000000000..254474910ca
--- /dev/null
+++ b/ludwig/schema/decoders/utils.py
@@ -0,0 +1,82 @@
+from dataclasses import field
+
+from marshmallow import fields, ValidationError
+
+from ludwig.constants import TYPE
+from ludwig.decoders.registry import get_decoder_classes, get_decoder_cls
+from ludwig.schema import utils as schema_utils
+
+
+def get_decoder_conds(feature_type: str):
+    """Returns a JSON schema of conditionals to validate against decoder types for specific feature types."""
+    conds = []
+    for decoder in get_decoder_classes(feature_type):
+        decoder_cls = get_decoder_cls(feature_type, decoder).get_schema_cls()
+        other_props = schema_utils.unload_jsonschema_from_marshmallow_class(decoder_cls)["properties"]
+        other_props.pop("type")
+        decoder_cond = schema_utils.create_cond(
+            {"type": decoder},
+            other_props,
+        )
+        conds.append(decoder_cond)
+    return conds
+
+
+def DecoderDataclassField(feature_type: str, default: str):
+    """Custom dataclass field that when used inside a dataclass will allow the user to specify a decoder config.
+
+    Returns: Initialized dataclass field that converts an untyped dict with params to a decoder config.
+    """
+
+    class DecoderMarshmallowField(fields.Field):
+        """Custom marshmallow field that deserializes a dict for a valid decoder config from the decoder_registry
+        and creates a corresponding `oneOf` JSON schema for external usage."""
+
+        def _deserialize(self, value, attr, data, **kwargs):
+            if value is None:
+                return None
+            if isinstance(value, dict):
+                if TYPE in value and value[TYPE] in get_decoder_classes(feature_type):
+                    dec = get_decoder_cls(feature_type, value[TYPE]).get_schema_cls()
+                    try:
+                        return dec.Schema().load(value)
+                    except (TypeError, ValidationError) as error:
+                        raise ValidationError(
+                            f"Invalid decoder params: {value}, see `{dec}` definition. Error: {error}"
+                        )
+                raise ValidationError(
+                    f"Invalid params for decoder: {value}, expect dict with at least a valid `type` attribute."
+                )
+            raise ValidationError("Field should be None or dict")
+
+        @staticmethod
+        def _jsonschema_type_mapping():
+            decoder_classes = list(get_decoder_classes(feature_type).keys())
+
+            return {
+                "type": "object",
+                "properties": {
+                    "type": {"type": "string", "enum": decoder_classes, "default": default},
+                },
+                "title": "decoder_options",
+                "allOf": get_decoder_conds(feature_type),
+                "required": ["type"],
+            }
+
+    try:
+        decoder = get_decoder_cls(feature_type, default).get_schema_cls()
+        load_default = decoder.Schema().load({"type": default})
+        dump_default = decoder.Schema().dump({"type": default})
+
+        return field(
+            metadata={
+                "marshmallow_field": DecoderMarshmallowField(
+                    allow_none=False,
+                    dump_default=dump_default,
+                    load_default=load_default,
+                )
+            },
+            default_factory=lambda: load_default,
+        )
+    except Exception as e:
+        raise ValidationError(f"Unsupported decoder type: {default}. See decoder_registry. " f"Details: {e}")
diff --git a/ludwig/schema/encoders/__init__.py b/ludwig/schema/encoders/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/ludwig/schema/encoders/bag_encoders.py b/ludwig/schema/encoders/bag_encoders.py
new file mode 100644
index 00000000000..2fbdd0e549c
--- /dev/null
+++ b/ludwig/schema/encoders/bag_encoders.py
@@ -0,0 +1,118 @@
+from typing import List
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+
+
+@dataclass
+class BagEmbedWeightedConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["embed"],
+        default="embed",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    vocab: List[str] = schema_utils.List(default=None, description="Vocabulary of the encoder")
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=50,
+        description="The maximum embedding size, the actual size will be min(vocabulary_size, embedding_size) for "
+        "dense representations and exactly vocabulary_size for the sparse encoding, where vocabulary_size "
+        "is the number of different strings appearing in the training set in the input column (plus 1 for "
+        "the unknown token placeholder <UNK>).",
+    )
+
+    representation: str = schema_utils.StringOptions(
+        ["dense", "sparse"],
+        default="dense",
+        description="The representation of the embedding. Either dense or sparse.",
+    )
+
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="If true embeddings are trained during the training process, if false embeddings are fixed. It "
+        "may be useful when loading pretrained embeddings for avoiding finetuning them. This parameter "
+        "has effect only when representation is dense as sparse one-hot encodings are not trainable.",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="By default dense embeddings are initialized randomly, but this parameter allows to specify a "
+        "path to a file containing embeddings in the GloVe format. When the file containing the "
+        "embeddings is loaded, only the embeddings with labels present in the vocabulary are kept, "
+        "the others are discarded. If the vocabulary contains strings that have no match in the "
+        "embeddings file, their embeddings are initialized with the average of all other embedding plus "
+        "some random noise to make them different from each other. This parameter has effect only if "
+        "representation is dense.",
+    )
+
+    force_embedding_size: bool = schema_utils.Boolean(
+        default=False,
+        description="Force the embedding size to be equal to the vocabulary size. This parameter has effect only if "
+        "representation is dense.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="By default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster "
+        "access, but in some cases the embedding matrix may be too large. This parameter forces the "
+        "placement of the embedding matrix in regular memory and the CPU is used for embedding lookup, "
+        "slightly slowing down the process as a result of data transfer between CPU and GPU memory.",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="This is the number of stacked fully connected layers that the input to the feature passes "
+        "through. Their output is projected in the feature's output space.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="If output_size is not already specified in fc_layers this is the default output_size that will "
+        "be used for each layer. It indicates the size of the output of a fully connected layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
diff --git a/ludwig/schema/encoders/base.py b/ludwig/schema/encoders/base.py
new file mode 100644
index 00000000000..3b9f3fee5ff
--- /dev/null
+++ b/ludwig/schema/encoders/base.py
@@ -0,0 +1,100 @@
+from abc import ABC
+from typing import List, Union
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+
+
+@dataclass
+class BaseEncoderConfig(schema_utils.BaseMarshmallowConfig, ABC):
+    """Base class for encoders.
+
+    Not meant to be used directly.
+    """
+
+    type: str
+    "Name corresponding to an encoder."
+
+
+@dataclass
+class PassthroughEncoderConfig(BaseEncoderConfig):
+    """PassthroughEncoderConfig is a dataclass that configures the parameters used for a passthrough encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["passthrough"],
+        default="passthrough",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+
+@dataclass
+class DenseEncoderConfig(BaseEncoderConfig):
+    """DenseEncoderConfig is a dataclass that configures the parameters used for a dense encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["dense"],
+        default="dense",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    input_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Size of the input to the dense encoder.",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(
+        default=None,
+        description="List of fully connected layers to use in the encoder.",
+    )
+
+    num_layers: int = schema_utils.PositiveInteger(
+        default=1,
+        description="Number of stacked fully connected layers that the input to the feature passes through.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Size of the output of the feature.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: Union[str, dict] = schema_utils.InitializerOptions(
+        description="Initializer for the weight matrix.",
+    )
+
+    bias_initializer: Union[str, dict] = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer for the bias vector.",
+    )
+
+    norm: Union[str] = schema_utils.StringOptions(
+        ["batch", "layer"],
+        allow_none=True,
+        default=None,
+        description="Normalization to use in the dense layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters for normalization if norm is either batch or layer.",
+    )
+
+    activation: str = schema_utils.StringOptions(
+        ["elu", "leakyRelu", "logSigmoid", "relu", "sigmoid", "tanh", "softmax"],
+        default="relu",
+        description="Activation function to apply to the output.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout rate.",
+    )
diff --git a/ludwig/schema/encoders/binary_encoders.py b/ludwig/schema/encoders/binary_encoders.py
new file mode 100644
index 00000000000..12de0c04d97
--- /dev/null
+++ b/ludwig/schema/encoders/binary_encoders.py
@@ -0,0 +1,15 @@
+from marshmallow_dataclass import dataclass
+
+import ludwig.schema.utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+
+
+@dataclass
+class BinaryPassthroughEncoderConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["passthrough"],
+        default="passthrough",
+        allow_none=False,
+        description="Type of encoder.",
+    )
diff --git a/ludwig/schema/encoders/category_encoders.py b/ludwig/schema/encoders/category_encoders.py
new file mode 100644
index 00000000000..8c69f1296b0
--- /dev/null
+++ b/ludwig/schema/encoders/category_encoders.py
@@ -0,0 +1,160 @@
+from typing import List
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+
+
+@dataclass
+class CategoricalEmbedConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["dense"],
+        default="dense",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    vocab: List[str] = schema_utils.List(
+        default=None,
+        description="Vocabulary of the encoder",
+    )
+
+    embedding_size: int = schema_utils.NonNegativeInteger(
+        default=50,
+        description="The maximum embedding size, the actual size will be min(vocabulary_size, embedding_size) for "
+        "dense representations and exactly vocabulary_size for the sparse encoding, where vocabulary_size "
+        "is the number of different strings appearing in the training set in the column the feature is "
+        "named after (plus 1 for <UNK>).",
+    )
+
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="If true embeddings are trained during the training process, if false embeddings are fixed. It "
+        "may be useful when loading pretrained embeddings for avoiding finetuning them. This parameter "
+        "has effect only when representation is dense as sparse one-hot encodings are not trainable. ",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="By default dense embeddings are initialized randomly, but this parameter allows to specify a "
+        "path to a file containing embeddings in the GloVe format. When the file containing the "
+        "embeddings is loaded, only the embeddings with labels present in the vocabulary are kept, "
+        "the others are discarded. If the vocabulary contains strings that have no match in the "
+        "embeddings file, their embeddings are initialized with the average of all other embedding plus "
+        "some random noise to make them different from each other. This parameter has effect only if "
+        "representation is dense.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="By default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster "
+        "access, but in some cases the embedding matrix may be too large. This parameter forces the "
+        "placement of the embedding matrix in regular memory and the CPU is used for embedding lookup, "
+        "slightly slowing down the process as a result of data transfer between CPU and GPU memory.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout rate.",
+    )
+
+    embedding_initializer: str = schema_utils.StringOptions(
+        [
+            "constant",
+            "identity",
+            "zeros",
+            "ones",
+            "orthogonal",
+            "normal",
+            "uniform",
+            "truncated_normal",
+            "variance_scaling",
+            "glorot_normal",
+            "glorot_uniform",
+            "xavier_normal",
+            "xavier_uniform",
+            "he_normal",
+            "he_uniform",
+            "lecun_normal",
+            "lecun_uniform",
+        ],
+        default=None,
+        description="Initializer for the embedding matrix.",
+    )
+
+
+@dataclass
+class CategoricalSparseConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["sparse"],
+        default="sparse",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    vocab: List[str] = schema_utils.List(
+        default=None,
+        description="Vocabulary of the encoder",
+    )
+
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="If true embeddings are trained during the training process, if false embeddings are fixed. It "
+        "may be useful when loading pretrained embeddings for avoiding finetuning them. This parameter "
+        "has effect only when representation is dense as sparse one-hot encodings are not trainable. ",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="By default dense embeddings are initialized randomly, but this parameter allows to specify a "
+        "path to a file containing embeddings in the GloVe format. When the file containing the "
+        "embeddings is loaded, only the embeddings with labels present in the vocabulary are kept, "
+        "the others are discarded. If the vocabulary contains strings that have no match in the "
+        "embeddings file, their embeddings are initialized with the average of all other embedding plus "
+        "some random noise to make them different from each other. This parameter has effect only if "
+        "representation is dense.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="By default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster "
+        "access, but in some cases the embedding matrix may be too large. This parameter forces the "
+        "placement of the embedding matrix in regular memory and the CPU is used for embedding lookup, "
+        "slightly slowing down the process as a result of data transfer between CPU and GPU memory.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout rate.",
+    )
+
+    embedding_initializer: str = schema_utils.StringOptions(
+        [
+            "constant",
+            "identity",
+            "zeros",
+            "ones",
+            "orthogonal",
+            "normal",
+            "uniform",
+            "truncated_normal",
+            "variance_scaling",
+            "glorot_normal",
+            "glorot_uniform",
+            "xavier_normal",
+            "xavier_uniform",
+            "he_normal",
+            "he_uniform",
+            "lecun_normal",
+            "lecun_uniform",
+        ],
+        default=None,
+        description="Initializer for the embedding matrix.",
+    )
diff --git a/ludwig/schema/encoders/date_encoders.py b/ludwig/schema/encoders/date_encoders.py
new file mode 100644
index 00000000000..075e3650a4d
--- /dev/null
+++ b/ludwig/schema/encoders/date_encoders.py
@@ -0,0 +1,145 @@
+from typing import List
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+
+
+@dataclass
+class DateEmbedConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["embed"],
+        default="embed",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="The maximum embedding size adopted.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to force the placement of the embedding matrix in regular memory and have the CPU "
+        "resolve them.",
+    )
+
+    # TODO (Connor): Add nesting logic for fc_layers, see fully_connected_module.py
+    fc_layers: List[dict] = schema_utils.DictList(
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="The number of stacked fully connected layers.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="If an output_size is not already specified in fc_layers this is the default output_size that "
+        "will be used for each layer. It indicates the size of the output of a fully connected layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
+
+
+@dataclass
+class DateWaveConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["wave"],
+        default="wave",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    # TODO (Connor): Add nesting logic for fc_layers, see fully_connected_module.py
+    fc_layers: List[dict] = schema_utils.DictList(
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.PositiveInteger(
+        default=1,
+        description="The number of stacked fully connected layers.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="If an output_size is not already specified in fc_layers this is the default output_size that "
+        "will be used for each layer. It indicates the size of the output of a fully connected layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
diff --git a/ludwig/schema/encoders/h3_encoders.py b/ludwig/schema/encoders/h3_encoders.py
new file mode 100644
index 00000000000..724f9208f05
--- /dev/null
+++ b/ludwig/schema/encoders/h3_encoders.py
@@ -0,0 +1,260 @@
+from typing import List
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+
+
+@dataclass
+class H3EmbedConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["embed"],
+        default="embed",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="The maximum embedding size adopted.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to force the placement of the embedding matrix in regular memory and have the CPU "
+        "resolve them.",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="The number of stacked fully connected layers.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="If an output_size is not already specified in fc_layers this is the default output_size that "
+        "will be used for each layer. It indicates the size of the output of a fully connected layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
+
+
+@dataclass
+class H3WeightedSumConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["weighted_sum"],
+        default="weighted_sum",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="The maximum embedding size adopted.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to force the placement of the embedding matrix in regular memory and have the CPU "
+        "resolve them.",
+    )
+
+    should_softmax: bool = schema_utils.Boolean(
+        default=False,
+        description="Determines if the weights of the weighted sum should be passed though a softmax layer before "
+        "being used.",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="The number of stacked fully connected layers.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="If an output_size is not already specified in fc_layers this is the default output_size that "
+        "will be used for each layer. It indicates the size of the output of a fully connected layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
+
+
+@dataclass
+class H3RNNConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["rnn"],
+        default="rnn",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="The maximum embedding size adopted.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to force the placement of the embedding matrix in regular memory and have the CPU "
+        "resolve them.",
+    )
+
+    num_layers: int = schema_utils.PositiveInteger(
+        default=1,
+        description="The number of stacked recurrent layers.",
+    )
+
+    hidden_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="The size of the hidden representation within the transformer block. It is usually the same as "
+        "the embedding_size, but if the two values are different, a projection layer will be added before "
+        "the first transformer block.",
+    )
+
+    cell_type: str = schema_utils.StringOptions(
+        ["rnn", "lstm", "lstm_block", "lstm", "ln", "lstm_cudnn", "gru", "gru_block", "gru_cudnn"],
+        default="rnn",
+        description="The type of recurrent cell to use. Available values are: `rnn`, `lstm`, `lstm_block`, `lstm`, "
+        "`ln`, `lstm_cudnn`, `gru`, `gru_block`, `gru_cudnn`. For reference about the differences between "
+        "the cells please refer to TensorFlow's documentation. We suggest to use the `block` variants on "
+        "CPU and the `cudnn` variants on GPU because of their increased speed. ",
+    )
+
+    bidirectional: bool = schema_utils.Boolean(
+        default=False,
+        description="If true, two recurrent networks will perform encoding in the forward and backward direction and "
+        "their outputs will be concatenated.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        default="tanh",
+        description="The activation function to use",
+    )
+
+    recurrent_activation: str = schema_utils.ActivationOptions(
+        default="sigmoid",
+        description="The activation function to use in the recurrent step",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use a bias vector.",
+    )
+
+    unit_forget_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="If true, add 1 to the bias of the forget gate at initialization",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    recurrent_initializer: str = schema_utils.InitializerOptions(
+        default="orthogonal", description="The initializer for recurrent matrix weights"
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    dropout: float = schema_utils.FloatRange(default=0.0, min=0, max=1, description="The dropout rate")
+
+    recurrent_dropout: float = schema_utils.FloatRange(
+        default=0.0, min=0, max=1, description="The dropout rate for the recurrent state"
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="last",
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
diff --git a/ludwig/schema/encoders/image_encoders.py b/ludwig/schema/encoders/image_encoders.py
new file mode 100644
index 00000000000..9ef2b604ffa
--- /dev/null
+++ b/ludwig/schema/encoders/image_encoders.py
@@ -0,0 +1,516 @@
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.utils.torch_utils import initializer_registry
+
+
+@dataclass
+class Stacked2DCNNEncoderConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["stacked_cnn"],
+        default="stacked_cnn",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    height: int = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Height of the input image.",
+    )
+
+    width: int = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Width of the input image.",
+    )
+
+    conv_layers: Optional[List[dict]] = schema_utils.DictList(
+        default=None,
+        description="List of convolutional layers to use in the encoder. ",
+    )
+
+    num_conv_layers: Optional[int] = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Number of convolutional layers to use in the encoder. ",
+    )
+
+    num_channels: Optional[int] = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Number of channels to use in the encoder. ",
+    )
+
+    out_channels: Optional[int] = schema_utils.NonNegativeInteger(
+        default=32,
+        description="Indicates the number of filters, and by consequence the output channels of the 2d convolution. "
+        "If out_channels is not already specified in conv_layers this is the default out_channels that "
+        "will be used for each layer. ",
+    )
+
+    kernel_size: Optional[Union[int, Tuple[int]]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=3,
+        description="An integer or pair of integers specifying the kernel size. A single integer specifies a square "
+        "kernel, while a pair of integers specifies the height and width of the kernel in that order (h, "
+        "w). If a kernel_size is not specified in conv_layers this kernel_size that will be used for "
+        "each layer.",
+    )
+
+    stride: Optional[Union[int, Tuple[int]]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=1,
+        description="An integer or pair of integers specifying the stride of the convolution along the height and "
+        "width. If a stride is not already specified in conv_layers, specifies the default stride of the "
+        "2D convolutional kernel that will be used for each layer. ",
+    )
+
+    padding: Optional[Union[int, Tuple[int], str]] = schema_utils.PositiveIntegerOrTupleOrStringOptions(
+        options=["valid", "same"],
+        default="valid",
+        description="An int, pair of ints (h, w), or one of valid, same specifying the padding used for convolution "
+        "kernels. ",
+    )
+
+    dilation: Optional[Union[int, Tuple[int]]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=1,
+        description="An int or pair of ints specifying the dilation rate to use for dilated convolution. If dilation "
+        "is not already specified in conv_layers, specifies the default dilation of the 2D convolutional "
+        "kernel that will be used for each layer.",
+    )
+
+    groups: Optional[int] = schema_utils.PositiveInteger(
+        default=1,
+        description="Groups controls the connectivity between convolution inputs and outputs. When groups = 1, each "
+        "output channel depends on every input channel. When groups > 1, input and output channels are "
+        "divided into groups separate groups, where each output channel depends only on the inputs in its "
+        "respective input channel group. in_channels and out_channels must both be divisible by groups.",
+    )
+
+    conv_use_bias: Optional[bool] = schema_utils.Boolean(
+        default=True,
+        description="If bias not already specified in conv_layers, specifies if the 2D convolutional kernel should "
+        "have a bias term.",
+    )
+
+    padding_mode: Optional[str] = schema_utils.StringOptions(
+        options=["zeros", "reflect", "replicate", "circular"],
+        default="zeros",
+        description="If padding_mode is not already specified in conv_layers, specifies the default padding_mode of "
+        "the 2D convolutional kernel that will be used for each layer.",
+    )
+
+    conv_norm: Optional[str] = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="If a norm is not already specified in conv_layers this is the default norm that will be used for "
+        "each layer. It indicates the normalization applied to the activations and can be null, "
+        "batch or layer.",
+    )
+
+    conv_norm_params: Optional[Dict[str, Any]] = schema_utils.Dict(
+        default=None,
+        description="Parameters used if conv_norm is either batch or layer. ",
+    )
+
+    conv_activation: str = schema_utils.ActivationOptions(
+        description="If an activation is not already specified in conv_layers this is the default activation that "
+        "will be used for each layer. It indicates the activation function applied to the output.",
+    )
+
+    conv_dropout: Optional[int] = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout rate",
+    )
+
+    pool_function: Optional[str] = schema_utils.StringOptions(
+        ["max", "average", "avg", "mean"],
+        default="max",
+        description="Pooling function to use.",
+    )
+
+    pool_kernel_size: Optional[Union[int, Tuple[int]]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=2,
+        description="An integer or pair of integers specifying the pooling size. If pool_kernel_size is not specified "
+        "in conv_layers this is the default value that will be used for each layer.",
+    )
+
+    pool_stride: Optional[Union[int, Tuple[int]]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=None,
+        description="An integer or pair of integers specifying the pooling stride, which is the factor by which the "
+        "pooling layer downsamples the feature map. Defaults to pool_kernel_size.",
+    )
+
+    pool_padding: Optional[Union[int, Tuple[int]]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=0,
+        description="An integer or pair of ints specifying pooling padding (h, w).",
+    )
+
+    pool_dilation: Optional[Union[int, Tuple[int]]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=1,
+        description="An integer or pair of ints specifying pooling dilation rate (h, w).",
+    )
+
+    fc_layers: Optional[Optional[List[Dict]]] = schema_utils.DictList(
+        default=None,
+        description="A list of dictionaries containing the parameters of all the fully connected layers. The length "
+        "of the list determines the number of stacked fully connected layers and the content of each "
+        "dictionary determines the parameters for a specific layer. The available parameters for each "
+        "layer are: activation, dropout, norm, norm_params, output_size, use_bias, bias_initializer and "
+        "weights_initializer. If any of those values is missing from the dictionary, the default one "
+        "specified as a parameter of the encoder will be used instead. ",
+    )
+
+    num_fc_layers: Optional[Optional[int]] = schema_utils.PositiveInteger(
+        default=1,
+        description="The number of stacked fully connected layers.",
+    )
+
+    output_size: Optional[int] = schema_utils.PositiveInteger(
+        default=128,
+        description="If output_size is not already specified in fc_layers this is the default output_size that will "
+        "be used for each layer. It indicates the size of the output of a fully connected layer. ",
+    )
+
+    fc_use_bias: Optional[bool] = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    fc_weights_initializer: Optional[str] = schema_utils.StringOptions(
+        sorted(list(initializer_registry.keys())),
+        default="xavier_uniform",
+        description="Initializer for the weights matrix.",
+    )
+
+    fc_bias_initializer: Optional[str] = schema_utils.StringOptions(
+        sorted(list(initializer_registry.keys())),
+        default="zeros",
+        description="Initializer for the bias vector.",
+    )
+
+    fc_norm: Optional[str] = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="If a norm is not already specified in fc_layers this is the default norm that will be used for "
+        "each layer. It indicates the norm of the output and can be null, batch or layer.",
+    )
+
+    fc_norm_params: Optional[Dict[str, Any]] = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either batch or layer. For information on parameters used with batch "
+        "see Torch's documentation on batch normalization or for layer see Torch's documentation on layer "
+        "normalization.",
+    )
+
+    fc_activation: Optional[str] = schema_utils.ActivationOptions(
+        description="If an activation is not already specified in fc_layers this is the default activation that will "
+        "be used for each layer. It indicates the activation function applied to the output.",
+    )
+
+    fc_dropout: Optional[float] = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout rate",
+    )
+
+
+@dataclass
+class ResNetEncoderConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["resnet"],
+        default="resnet",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    height: int = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Height of the input image.",
+    )
+
+    width: int = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Width of the input image.",
+    )
+
+    resnet_size: Optional[int] = schema_utils.PositiveInteger(
+        default=50,
+        description="The size of the ResNet model to use.",
+    )
+
+    num_channels: Optional[int] = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Number of channels to use in the encoder. ",
+    )
+
+    out_channels: Optional[int] = schema_utils.NonNegativeInteger(
+        default=32,
+        description="Indicates the number of filters, and by consequence the output channels of the 2d convolution. "
+        "If out_channels is not already specified in conv_layers this is the default out_channels that "
+        "will be used for each layer. ",
+    )
+
+    kernel_size: Optional[Union[int, Tuple[int]]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=3,
+        description="An integer or pair of integers specifying the kernel size. A single integer specifies a square "
+        "kernel, while a pair of integers specifies the height and width of the kernel in that order (h, "
+        "w). If a kernel_size is not specified in conv_layers this kernel_size that will be used for "
+        "each layer.",
+    )
+
+    conv_stride: Union[int, Tuple[int]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=1,
+        description="An integer or pair of integers specifying the stride of the initial convolutional layer.",
+    )
+
+    first_pool_kernel_size: Union[int, Tuple[int]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=None,
+        description="Pool size to be used for the first pooling layer. If none, the first pooling layer is skipped.",
+    )
+
+    first_pool_stride: Union[int, Tuple[int]] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=None,
+        description="Stride for first pooling layer. If null, defaults to first_pool_kernel_size.",
+    )
+
+    batch_norm_momentum: float = schema_utils.NonNegativeFloat(
+        default=0.9,
+        description="Momentum of the batch norm running statistics.",
+    )
+
+    batch_norm_epsilon: float = schema_utils.NonNegativeFloat(
+        default=0.001,
+        description="Epsilon of the batch norm.",
+    )
+
+    fc_layers: Optional[Optional[List[Dict]]] = schema_utils.DictList(
+        default=None,
+        description="A list of dictionaries containing the parameters of all the fully connected layers. The length "
+        "of the list determines the number of stacked fully connected layers and the content of each "
+        "dictionary determines the parameters for a specific layer. The available parameters for each "
+        "layer are: activation, dropout, norm, norm_params, output_size, use_bias, bias_initializer and "
+        "weights_initializer. If any of those values is missing from the dictionary, the default one "
+        "specified as a parameter of the encoder will be used instead. ",
+    )
+
+    num_fc_layers: Optional[Optional[int]] = schema_utils.PositiveInteger(
+        default=1,
+        description="The number of stacked fully connected layers.",
+    )
+
+    output_size: Optional[int] = schema_utils.PositiveInteger(
+        default=128,
+        description="if output_size is not already specified in fc_layers this is the default output_size that will "
+        "be used for each layer. It indicates the size of the output of a fully connected layer. ",
+    )
+
+    use_bias: Optional[bool] = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: Optional[str] = schema_utils.StringOptions(
+        sorted(list(initializer_registry.keys())),
+        default="xavier_uniform",
+        description="Initializer for the weights matrix.",
+    )
+
+    bias_initializer: Optional[str] = schema_utils.StringOptions(
+        sorted(list(initializer_registry.keys())),
+        default="zeros",
+        description="initializer for the bias vector.",
+    )
+
+    norm: Optional[str] = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="if a norm is not already specified in fc_layers this is the default norm that will be used for "
+        "each layer. It indicates the norm of the output and can be null, batch or layer.",
+    )
+
+    norm_params: Optional[Dict[str, Any]] = schema_utils.Dict(
+        default=None,
+        description="parameters used if norm is either batch or layer. For information on parameters used with batch "
+        "see Torch's documentation on batch normalization or for layer see Torch's documentation on layer "
+        "normalization.",
+    )
+
+    activation: Optional[str] = schema_utils.ActivationOptions(
+        description="if an activation is not already specified in fc_layers this is the default activation that will "
+        "be used for each layer. It indicates the activation function applied to the output.",
+    )
+
+    dropout: Optional[float] = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout rate",
+    )
+
+
+@dataclass
+class MLPMixerEncoderConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["mlp_mixer"],
+        default="mlp_mixer",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    height: int = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Height of the input image.",
+    )
+
+    width: int = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Width of the input image.",
+    )
+
+    num_channels: int = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Number of channels to use in the encoder. ",
+    )
+
+    patch_size: int = schema_utils.PositiveInteger(
+        default=16,
+        description="The image patch size. Each patch is patch_size² pixels. Must evenly divide the image width and "
+        "height.",
+    )
+
+    embed_size: int = schema_utils.PositiveInteger(
+        default=512,
+        description="The patch embedding size, the output size of the mixer if avg_pool is true.",
+    )
+
+    token_size: int = schema_utils.PositiveInteger(
+        default=2048,
+        description="The per-patch embedding size.",
+    )
+
+    channel_dim: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Number of channels in hidden layer.",
+    )
+
+    num_layers: int = schema_utils.PositiveInteger(
+        default=8,
+        description="The depth of the network (the number of Mixer blocks).",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout rate.",
+    )
+
+    avg_pool: bool = schema_utils.Boolean(
+        default=True,
+        description="If true, pools output over patch dimension, outputs a vector of shape (embed_size). If false, "
+        "the output tensor is of shape (n_patches, embed_size), where n_patches is img_height x img_width "
+        "/ patch_size².",
+    )
+
+
+@dataclass
+class ViTEncoderConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["vit"],
+        default="vit",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    height: int = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Height of the input image.",
+    )
+
+    width: int = schema_utils.NonNegativeInteger(
+        default=None,
+        description="Width of the input image.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Use pre-trained model weights from Hugging Face.",
+    )
+
+    pretrained_model: str = schema_utils.String(
+        default="google/vit-base-patch16-224",
+        description="The name of the pre-trained model to use.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="",
+    )
+
+    hidden_size: int = schema_utils.PositiveInteger(
+        default=768,
+        description="Dimensionality of the encoder layers and the pooling layer.",
+    )
+
+    num_hidden_layers: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    num_attention_heads: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of attention heads in each attention layer.",
+    )
+
+    intermediate_size: int = schema_utils.PositiveInteger(
+        default=3072,
+        description="Dimensionality of the intermediate (i.e., feed-forward) layer in the Transformer encoder.",
+    )
+
+    hidden_act: str = schema_utils.StringOptions(
+        ["relu", "gelu", "selu", "gelu_new"],
+        default="gelu",
+        description="Hidden layer activation, one of gelu, relu, selu or gelu_new.",
+    )
+
+    hidden_dropout_prob: float = schema_utils.NonNegativeFloat(
+        default=0.1,
+        description="The dropout rate for all fully connected layers in the embeddings, encoder, and pooling.",
+    )
+
+    attention_probs_dropout_prob: float = schema_utils.NonNegativeFloat(
+        default=0.1,
+        description="The dropout rate for the attention probabilities.",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    layer_norm_eps: float = schema_utils.NonNegativeFloat(
+        default=1e-12,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    gradient_checkpointing: bool = schema_utils.Boolean(
+        default=False,
+        description="",
+    )
+
+    patch_size: int = schema_utils.PositiveInteger(
+        default=16,
+        description="The image patch size. Each patch is patch_size² pixels. Must evenly divide the image width and "
+        "height.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="Is the encoder trainable.",
+    )
diff --git a/ludwig/schema/encoders/sequence_encoders.py b/ludwig/schema/encoders/sequence_encoders.py
new file mode 100644
index 00000000000..7edf0963864
--- /dev/null
+++ b/ludwig/schema/encoders/sequence_encoders.py
@@ -0,0 +1,1090 @@
+from typing import List
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+
+
+@dataclass
+class SequencePassthroughConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["passthrough"],
+        default="passthrough",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default=None,
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The maximum length of a sequence.",
+    )
+
+    encoding_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="The size of the encoding vector, or None if sequence elements are scalars.",
+    )
+
+
+@dataclass
+class SequenceEmbedConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["embed"],
+        default="embed",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="The maximum length of a sequence.",
+    )
+
+    representation: str = schema_utils.StringOptions(
+        ["dense", "sparse"],
+        default="dense",
+        description="Representation of the embedding.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Size of the embedding.",
+    )
+
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the embedding is trainable.",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="Path to a file containing pretrained embeddings.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to force the placement of the embedding matrix in regular memory and have the CPU "
+        "resolve them.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        default="uniform",
+        description="Initializer to use for the weights matrix.",
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
+
+
+@dataclass
+class ParallelCNNConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["parallel_cnn"],
+        default="parallel_cnn",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    should_embed: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to embed the input sequence.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    representation: str = schema_utils.StringOptions(
+        ["dense", "sparse"],
+        default="dense",
+        description="Representation of the embedding.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Size of the embedding.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None, description="The maximum length of all sequences"
+    )
+
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the embedding is trainable.",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="Path to a file containing pretrained embeddings.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to force the placement of the embedding matrix in regular memory and have the CPU "
+        "resolve them.",
+    )
+
+    conv_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for conv_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each convolutional layer.",
+    )
+
+    num_conv_layers: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of parallel convolutional layers to use.",
+    )
+
+    filter_size: int = schema_utils.PositiveInteger(
+        default=3,
+        description="Size of the 1d convolutional filter.",
+    )
+
+    num_filters: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Number of filters, and by consequence number of output channels of the 1d convolution.",
+    )
+
+    pool_function: str = schema_utils.ReductionOptions(
+        default="max",
+        description="Pooling function to use.",
+    )
+
+    pool_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="The default pool_size that will be used for each layer.",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of parallel fully connected layers to use.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The default output_size that will be used for each layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
+
+
+@dataclass
+class StackedCNNConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["stacked_cnn"],
+        default="stacked_cnn",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    should_embed: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to embed the input sequence.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    representation: str = schema_utils.StringOptions(
+        ["dense", "sparse"],
+        default="dense",
+        description="Representation of the embedding.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Size of the embedding.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None, description="The maximum length of all sequences"
+    )
+
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the embedding is trainable.",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="Path to a file containing pretrained embeddings.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to force the placement of the embedding matrix in regular memory and have the CPU "
+        "resolve them.",
+    )
+
+    conv_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for conv_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each convolutional layer.",
+    )
+
+    num_conv_layers: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of parallel convolutional layers to use.",
+    )
+
+    filter_size: int = schema_utils.PositiveInteger(
+        default=3,
+        description="Size of the 1d convolutional filter.",
+    )
+
+    num_filters: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Number of filters, and by consequence number of output channels of the 1d convolution.",
+    )
+
+    strides: int = schema_utils.PositiveInteger(
+        default=1,
+        description="Stride length of the convolution.",
+    )
+
+    padding: str = schema_utils.StringOptions(
+        ["valid", "same"],
+        default="same",
+        description="Padding to use.",
+    )
+
+    dilation_rate: int = schema_utils.PositiveInteger(
+        default=1,
+        description="Dilation rate to use for dilated convolution.",
+    )
+
+    pool_function: str = schema_utils.ReductionOptions(
+        default="max",
+        description="Pooling function to use.",
+    )
+
+    pool_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="The default pool_size that will be used for each layer.",
+    )
+
+    pool_strides: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Factor to scale down.",
+    )
+
+    pool_padding: str = schema_utils.StringOptions(
+        ["valid", "same"],
+        default="same",
+        description="Padding to use.",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of parallel fully connected layers to use.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The default output_size that will be used for each layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
+
+
+@dataclass
+class StackedParallelCNNConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["stacked_parallel_cnn"],
+        default="stacked_parallel_cnn",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    should_embed: bool = schema_utils.Boolean(
+        default=True,
+        description="If True the input sequence is expected to be made of integers and will be mapped into embeddings",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary of the input feature to encode",
+    )
+
+    representation: str = schema_utils.StringOptions(
+        ["dense", "sparse"],
+        default="dense",
+        description="The representation of the embeddings. 'Dense' means the embeddings are initialized randomly. "
+        "'Sparse' means they are initialized to be one-hot encodings.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The maximum embedding size. The actual size will be `min(vocabulary_size, embedding_size)` for "
+        "`dense` representations and exactly `vocabulary_size` for the `sparse` encoding, "
+        "where `vocabulary_size` is the number of different strings appearing in the training set in the "
+        "column the feature is named after (plus 1 for `<UNK>`).",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None, description="The maximum length of all sequences"
+    )
+
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="If true embeddings are trained during the training process, if false embeddings are fixed. It "
+        "may be useful when loading pretrained embeddings for avoiding finetuning them. This parameter "
+        "has effect only when representation is dense as sparse one-hot encodings are not trainable. ",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="By default dense embeddings are initialized randomly, but this parameter allows to specify a "
+        "path to a file containing embeddings in the GloVe format. When the file containing the "
+        "embeddings is loaded, only the embeddings with labels present in the vocabulary are kept, "
+        "the others are discarded. If the vocabulary contains strings that have no match in the "
+        "embeddings file, their embeddings are initialized with the average of all other embedding plus "
+        "some random noise to make them different from each other. This parameter has effect only if "
+        "representation is dense.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="by default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster "
+        "access, but in some cases the embedding matrix may be too large. This parameter forces the "
+        "placement of the embedding matrix in regular memory and the CPU is used for embedding lookup, "
+        "slightly slowing down the process as a result of data transfer between CPU and GPU memory.",
+    )
+
+    stacked_layers: List[dict] = schema_utils.DictList(
+        default=None,
+        description="a nested list of lists of dictionaries containing the parameters of the stack of parallel "
+        "convolutional layers. The length of the list determines the number of stacked parallel "
+        "convolutional layers, length of the sub-lists determines the number of parallel conv layers and "
+        "the content of each dictionary determines the parameters for a specific layer. ",
+    )
+
+    num_stacked_layers: int = schema_utils.PositiveInteger(
+        default=None,
+        description="If stacked_layers is null, this is the number of elements in the stack of parallel convolutional "
+        "layers. ",
+    )
+
+    filter_size: int = schema_utils.PositiveInteger(
+        default=3,
+        description="Size of the 1d convolutional filter.",
+    )
+
+    num_filters: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Number of filters, and by consequence number of output channels of the 1d convolution.",
+    )
+
+    pool_function: str = schema_utils.ReductionOptions(
+        default="max",
+        description="Pooling function to use.",
+    )
+
+    pool_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="The default pool_size that will be used for each layer.",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of parallel fully connected layers to use.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The default output_size that will be used for each layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
+
+
+@dataclass
+class StackedRNNConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["rnn"],
+        default="rnn",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    should_embed: bool = schema_utils.Boolean(
+        default=True,
+        description="If True the input sequence is expected to be made of integers and will be mapped into embeddings",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary of the input feature to encode",
+    )
+
+    representation: str = schema_utils.StringOptions(
+        ["dense", "sparse"],
+        default="dense",
+        description="The representation of the embeddings. 'Dense' means the embeddings are initialized randomly. "
+        "'Sparse' means they are initialized to be one-hot encodings.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The maximum embedding size. The actual size will be `min(vocabulary_size, embedding_size)` for "
+        "`dense` representations and exactly `vocabulary_size` for the `sparse` encoding, "
+        "where `vocabulary_size` is the number of different strings appearing in the training set in the "
+        "column the feature is named after (plus 1 for `<UNK>`).",
+    )
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="If true embeddings are trained during the training process, if false embeddings are fixed. It "
+        "may be useful when loading pretrained embeddings for avoiding finetuning them. This parameter "
+        "has effect only when representation is dense as sparse one-hot encodings are not trainable. ",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="By default dense embeddings are initialized randomly, but this parameter allows to specify a "
+        "path to a file containing embeddings in the GloVe format. When the file containing the "
+        "embeddings is loaded, only the embeddings with labels present in the vocabulary are kept, "
+        "the others are discarded. If the vocabulary contains strings that have no match in the "
+        "embeddings file, their embeddings are initialized with the average of all other embedding plus "
+        "some random noise to make them different from each other. This parameter has effect only if "
+        "representation is dense.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="by default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster "
+        "access, but in some cases the embedding matrix may be too large. This parameter forces the "
+        "placement of the embedding matrix in regular memory and the CPU is used for embedding lookup, "
+        "slightly slowing down the process as a result of data transfer between CPU and GPU memory.",
+    )
+
+    num_layers: int = schema_utils.PositiveInteger(default=1, description="the number of stacked recurrent layers.")
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None, description="The maximum length of all sequences"
+    )
+
+    state_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The size of the state of the rnn.",
+    )
+
+    cell_type: str = schema_utils.StringOptions(
+        ["rnn", "lstm", "lstm_block", "lstm", "ln", "lstm_cudnn", "gru", "gru_block", "gru_cudnn"],
+        default="rnn",
+        description="The type of recurrent cell to use. Available values are: `rnn`, `lstm`, `lstm_block`, `lstm`, "
+        "`ln`, `lstm_cudnn`, `gru`, `gru_block`, `gru_cudnn`. For reference about the differences between "
+        "the cells please refer to TensorFlow's documentation. We suggest to use the `block` variants on "
+        "CPU and the `cudnn` variants on GPU because of their increased speed. ",
+    )
+
+    bidirectional: bool = schema_utils.Boolean(
+        default=False,
+        description="If true, two recurrent networks will perform encoding in the forward and backward direction and "
+        "their outputs will be concatenated.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        default="tanh",
+        description="The activation function to use",
+    )
+
+    recurrent_activation: str = schema_utils.ActivationOptions(
+        default="sigmoid",
+        description="The activation function to use in the recurrent step",
+    )
+
+    unit_forget_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="If true, add 1 to the bias of the forget gate at initialization",
+    )
+
+    recurrent_initializer: str = schema_utils.InitializerOptions(
+        default="orthogonal", description="The initializer for recurrent matrix weights"
+    )
+
+    dropout: float = schema_utils.FloatRange(default=0.0, min=0, max=1, description="The dropout rate")
+
+    recurrent_dropout: float = schema_utils.FloatRange(
+        default=0.0, min=0, max=1, description="The dropout rate for the recurrent state"
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="Number of parallel fully connected layers to use.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The default output_size that will be used for each layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    fc_activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each fully connected layer."
+    )
+
+    fc_dropout: float = schema_utils.FloatRange(
+        default=0.0, min=0, max=1, description="The dropout rate for fully connected layers"
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="last",
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
+
+
+@dataclass
+class StackedCNNRNNConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["cnnrnn"],
+        default="cnnrnn",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    should_embed: bool = schema_utils.Boolean(
+        default=True,
+        description="If True the input sequence is expected to be made of integers and will be mapped into embeddings",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary of the input feature to encode",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None, description="The maximum length of all sequences"
+    )
+
+    representation: str = schema_utils.StringOptions(
+        ["dense", "sparse"],
+        default="dense",
+        description="The representation of the embeddings. 'Dense' means the embeddings are initialized randomly. "
+        "'Sparse' means they are initialized to be one-hot encodings.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The maximum embedding size. The actual size will be `min(vocabulary_size, embedding_size)` for "
+        "`dense` representations and exactly `vocabulary_size` for the `sparse` encoding, "
+        "where `vocabulary_size` is the number of different strings appearing in the training set in the "
+        "column the feature is named after (plus 1 for `<UNK>`).",
+    )
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="If true embeddings are trained during the training process, if false embeddings are fixed. It "
+        "may be useful when loading pretrained embeddings for avoiding finetuning them. This parameter "
+        "has effect only when representation is dense as sparse one-hot encodings are not trainable. ",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="By default dense embeddings are initialized randomly, but this parameter allows to specify a "
+        "path to a file containing embeddings in the GloVe format. When the file containing the "
+        "embeddings is loaded, only the embeddings with labels present in the vocabulary are kept, "
+        "the others are discarded. If the vocabulary contains strings that have no match in the "
+        "embeddings file, their embeddings are initialized with the average of all other embedding plus "
+        "some random noise to make them different from each other. This parameter has effect only if "
+        "representation is dense.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="By default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster "
+        "access, but in some cases the embedding matrix may be too large. This parameter forces the "
+        "placement of the embedding matrix in regular memory and the CPU is used for embedding lookup, "
+        "slightly slowing down the process as a result of data transfer between CPU and GPU memory.",
+    )
+
+    conv_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for conv_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each convolutional layer.",
+    )
+
+    num_conv_layers: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of parallel convolutional layers to use.",
+    )
+
+    num_filters: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Number of filters, and by consequence number of output channels of the 1d convolution.",
+    )
+
+    filter_size: int = schema_utils.PositiveInteger(
+        default=5,
+        description="Size of the 1d convolutional filter.",
+    )
+
+    strides: int = schema_utils.PositiveInteger(
+        default=1,
+        description="Stride length of the convolution.",
+    )
+
+    padding: str = schema_utils.StringOptions(
+        ["valid", "same"],
+        default="same",
+        description="Padding to use.",
+    )
+
+    dilation_rate: int = schema_utils.PositiveInteger(
+        default=1,
+        description="Dilation rate to use for dilated convolution.",
+    )
+
+    conv_activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each convolutional layer."
+    )
+
+    conv_dropout: float = schema_utils.FloatRange(
+        default=0.0, min=0, max=1, description="The dropout rate for the convolutional layers"
+    )
+
+    pool_function: str = schema_utils.ReductionOptions(
+        default="max",
+        description="Pooling function to use.",
+    )
+
+    pool_size: int = schema_utils.PositiveInteger(
+        default=2,
+        description="The default pool_size that will be used for each layer.",
+    )
+
+    pool_strides: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Factor to scale down.",
+    )
+
+    pool_padding: str = schema_utils.StringOptions(
+        ["valid", "same"],
+        default="same",
+        description="Padding to use.",
+    )
+
+    num_rec_layers: int = schema_utils.PositiveInteger(default=1, description="The number of stacked recurrent layers.")
+
+    state_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The size of the state of the rnn.",
+    )
+
+    cell_type: str = schema_utils.StringOptions(
+        ["rnn", "lstm", "lstm_block", "lstm", "ln", "lstm_cudnn", "gru", "gru_block", "gru_cudnn"],
+        default="rnn",
+        description="The type of recurrent cell to use. Available values are: `rnn`, `lstm`, `lstm_block`, `lstm`, "
+        "`ln`, `lstm_cudnn`, `gru`, `gru_block`, `gru_cudnn`. For reference about the differences between "
+        "the cells please refer to TensorFlow's documentation. We suggest to use the `block` variants on "
+        "CPU and the `cudnn` variants on GPU because of their increased speed. ",
+    )
+
+    bidirectional: bool = schema_utils.Boolean(
+        default=False,
+        description="If true, two recurrent networks will perform encoding in the forward and backward direction and "
+        "their outputs will be concatenated.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        default="tanh",
+        description="The activation function to use",
+    )
+
+    recurrent_activation: str = schema_utils.ActivationOptions(
+        default="sigmoid",
+        description="The activation function to use in the recurrent step",
+    )
+
+    unit_forget_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="If true, add 1 to the bias of the forget gate at initialization",
+    )
+
+    recurrent_initializer: str = schema_utils.InitializerOptions(
+        default="orthogonal", description="The initializer for recurrent matrix weights"
+    )
+
+    dropout: float = schema_utils.FloatRange(default=0.0, min=0, max=1, description="The dropout rate")
+
+    recurrent_dropout: float = schema_utils.FloatRange(
+        default=0.0, min=0, max=1, description="The dropout rate for the recurrent state"
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="Number of parallel fully connected layers to use.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The default output_size that will be used for each layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    fc_activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each fully connected layer."
+    )
+
+    fc_dropout: float = schema_utils.FloatRange(
+        default=0.0, min=0, max=1, description="The dropout rate for fully connected layers"
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="last",
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
+
+
+@dataclass
+class StackedTransformerConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["transformer"],
+        default="transformer",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(default=None, description="Max length of all sequences")
+
+    should_embed: bool = schema_utils.Boolean(
+        default=True,
+        description="If True the input sequence is expected to be made of integers and will be mapped into embeddings",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary of the input feature to encode",
+    )
+
+    representation: str = schema_utils.StringOptions(
+        ["dense", "sparse"],
+        default="dense",
+        description="The representation of the embeddings. 'Dense' means the embeddings are initialized randomly. "
+        "'Sparse' means they are initialized to be one-hot encodings.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Size of the embedding.",
+    )
+
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the embedding is trainable.",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="Path to a file containing pretrained embeddings.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to force the placement of the embedding matrix in regular memory and have the CPU "
+        "resolve them.",
+    )
+
+    num_layers: int = schema_utils.PositiveInteger(default=1, description="the number of stacked recurrent layers.")
+
+    hidden_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The size of the hidden representation within the transformer block. It is usually the same as "
+        "the embedding_size, but if the two values are different, a projection layer will be added before "
+        "the first transformer block.",
+    )
+
+    num_heads: int = schema_utils.PositiveInteger(
+        default=8,
+        description="Number of attention heads in each transformer block.",
+    )
+
+    transformer_output_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Size of the fully connected layer after self attention in the transformer block. This is usually "
+        "the same as hidden_size and embedding_size.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout rate for the transformer block",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="Number of parallel fully connected layers to use.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The default output_size that will be used for each layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    fc_activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each fully connected layer."
+    )
+
+    fc_dropout: float = schema_utils.FloatRange(
+        default=0.0, min=0, max=1, description="The dropout rate for fully connected layers"
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="last",
+        description="How to reduce the output tensor along the `s` sequence length dimension if the rank of the "
+        "tensor is greater than 2.",
+    )
diff --git a/ludwig/schema/encoders/set_encoders.py b/ludwig/schema/encoders/set_encoders.py
new file mode 100644
index 00000000000..0a4ad52a5d7
--- /dev/null
+++ b/ludwig/schema/encoders/set_encoders.py
@@ -0,0 +1,112 @@
+from typing import List
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+
+
+@dataclass
+class SetSparseEncoderConfig(BaseEncoderConfig):
+
+    type: str = schema_utils.StringOptions(
+        ["embed"],
+        default="embed",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    vocab: List[str] = schema_utils.List(default=None, description="Vocabulary of the encoder")
+
+    representation: str = schema_utils.StringOptions(
+        ["dense", "sparse"],
+        default="dense",
+        description="The representation of the embedding. Either dense or sparse.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=50,
+        description="The maximum embedding size, the actual size will be min(vocabulary_size, embedding_size) for "
+        "dense representations and exactly vocabulary_size for the sparse encoding, where vocabulary_size "
+        "is the number of different strings appearing in the training set in the input column (plus 1 for "
+        "the unknown token placeholder <UNK>).",
+    )
+
+    embeddings_trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="If true embeddings are trained during the training process, if false embeddings are fixed. It "
+        "may be useful when loading pretrained embeddings for avoiding finetuning them. This parameter "
+        "has effect only when representation is dense as sparse one-hot encodings are not trainable.",
+    )
+
+    pretrained_embeddings: str = schema_utils.String(
+        default=None,
+        description="By default dense embeddings are initialized randomly, but this parameter allows to specify a "
+        "path to a file containing embeddings in the GloVe format. When the file containing the "
+        "embeddings is loaded, only the embeddings with labels present in the vocabulary are kept, "
+        "the others are discarded. If the vocabulary contains strings that have no match in the "
+        "embeddings file, their embeddings are initialized with the average of all other embedding plus "
+        "some random noise to make them different from each other. This parameter has effect only if "
+        "representation is dense.",
+    )
+
+    embeddings_on_cpu: bool = schema_utils.Boolean(
+        default=False,
+        description="By default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster "
+        "access, but in some cases the embedding matrix may be too large. This parameter forces the "
+        "placement of the embedding matrix in regular memory and the CPU is used for embedding lookup, "
+        "slightly slowing down the process as a result of data transfer between CPU and GPU memory.",
+    )
+
+    fc_layers: List[dict] = schema_utils.DictList(  # TODO (Connor): Add nesting logic for fc_layers
+        default=None,
+        description="List of dictionaries containing the parameters for each fully connected layer.",
+    )
+
+    num_fc_layers: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="This is the number of stacked fully connected layers that the input to the feature passes "
+        "through. Their output is projected in the feature's output space.",
+    )
+
+    output_size: int = schema_utils.PositiveInteger(
+        default=10,
+        description="If output_size is not already specified in fc_layers this is the default output_size that will "
+        "be used for each layer. It indicates the size of the output of a fully connected layer.",
+    )
+
+    use_bias: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether the layer uses a bias vector.",
+    )
+
+    weights_initializer: str = schema_utils.InitializerOptions(
+        description="Initializer to use for the weights matrix.",
+    )
+
+    bias_initializer: str = schema_utils.InitializerOptions(
+        default="zeros",
+        description="Initializer to use for the bias vector.",
+    )
+
+    norm: str = schema_utils.StringOptions(
+        ["batch", "layer"],
+        default=None,
+        description="The default norm that will be used for each layer.",
+    )
+
+    norm_params: dict = schema_utils.Dict(
+        default=None,
+        description="Parameters used if norm is either `batch` or `layer`.",
+    )
+
+    activation: str = schema_utils.ActivationOptions(
+        description="The default activation function that will be used for each layer."
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Dropout probability for the embedding.",
+    )
diff --git a/ludwig/schema/encoders/text_encoders.py b/ludwig/schema/encoders/text_encoders.py
new file mode 100644
index 00000000000..51a9640b49a
--- /dev/null
+++ b/ludwig/schema/encoders/text_encoders.py
@@ -0,0 +1,2400 @@
+from typing import Callable, List, Union
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+
+
+@dataclass
+class ALBERTConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an ALBERT encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["albert"],
+        default="albert",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="albert-base-v2",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="cls_pooled",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=30000,
+        description="Vocabulary size of the ALBERT model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=128,
+        description="Dimensionality of vocabulary embeddings.",
+    )
+
+    hidden_size: int = schema_utils.PositiveInteger(
+        default=4096,
+        description="Dimensionality of the encoder layers and the pooler layer.",
+    )
+
+    num_hidden_layers: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    num_hidden_groups: int = schema_utils.PositiveInteger(
+        default=1,
+        description="Number of groups for the hidden layers, parameters in the same group are shared.",
+    )
+
+    num_attention_heads: int = schema_utils.PositiveInteger(
+        default=64,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    intermediate_size: int = schema_utils.PositiveInteger(
+        default=16384,
+        description="The dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer "
+        "encoder.",
+    )
+
+    inner_group_num: int = schema_utils.PositiveInteger(
+        default=1,
+        description="The number of inner repetition of attention and ffn.",
+    )
+
+    hidden_act: str = schema_utils.StringOptions(
+        ["gelu", "relu", "silu", "gelu_new"],
+        default="gelu_new",
+        description="The non-linear activation function (function or string) in the encoder and pooler.",
+    )
+
+    hidden_dropout_prob: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    attention_probs_dropout_prob: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="The dropout ratio for the attention probabilities.",
+    )
+
+    max_position_embeddings: int = schema_utils.PositiveInteger(
+        default=512,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large (e.g., 512 or 1024 or 2048).",
+    )
+
+    type_vocab_size: int = schema_utils.PositiveInteger(
+        default=2,
+        description="The vocabulary size of the token_type_ids passed when calling AlbertModel or TFAlbertModel.",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    layer_norm_eps: float = schema_utils.NonNegativeFloat(
+        default=1e-12,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    classifier_dropout_prob: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout ratio for attached classifiers.",
+    )
+
+    position_embedding_type: str = schema_utils.StringOptions(
+        ["absolute", "relative_key", "relative_key_query"],
+        default="absolute",
+        description="",
+    )
+
+    pad_token_id: int = schema_utils.Integer(
+        default=0,
+        description="The ID of the token to use as padding.",
+    )
+
+    bos_token_id: int = schema_utils.Integer(
+        default=2,
+        description="The beginning of sequence token ID.",
+    )
+
+    eos_token_id: int = schema_utils.Integer(
+        default=3,
+        description="The end of sequence token ID.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class MT5Config(BaseEncoderConfig):
+    """This dataclass configures the schema used for an MT5 encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["mt5"],
+        default="mt5",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="google/mt5-base",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="cls_pooled",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=250112,
+        description="Vocabulary size of the T5 model. Defines the number of different tokens that can be represented "
+        "by the inputs_ids passed when calling T5Model or TFT5Model.",
+    )
+
+    d_model: int = schema_utils.PositiveInteger(
+        default=512,
+        description="Size of the encoder layers and the pooler layer.",
+    )
+
+    d_kv: int = schema_utils.PositiveInteger(
+        default=64,
+        description="Size of the key, query, value projections per attention head. d_kv has to be equal to d_model // "
+        "num_heads.",
+    )
+
+    d_ff: int = schema_utils.PositiveInteger(
+        default=1024,
+        description="Size of the intermediate feed forward layer in each T5Block.",
+    )
+
+    num_layers: int = schema_utils.PositiveInteger(
+        default=8,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    num_decoder_layers: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of hidden layers in the Transformer decoder. Will use the same value as num_layers if not "
+        "set.",
+    )
+
+    num_heads: int = schema_utils.PositiveInteger(
+        default=6,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    relative_attention_num_buckets: int = schema_utils.PositiveInteger(
+        default=32,
+        description="The number of buckets to use for each attention layer.",
+    )
+
+    dropout_rate: float = schema_utils.NonNegativeFloat(
+        default=0.1,
+        description="The ratio for all dropout layers.",
+    )
+
+    layer_norm_epsilon: float = schema_utils.NonNegativeFloat(
+        default=1e-06,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    initializer_factor: float = schema_utils.NonNegativeFloat(
+        default=1.0,
+        description="A factor for initializing all weight matrices (should be kept to 1, used internally for "
+        "initialization testing)",
+    )
+
+    feed_forward_proj: str = schema_utils.StringOptions(
+        ["relu", "gated-gelu"],
+        default="gated-gelu",
+        description="Type of feed forward layer to be used. ",
+    )
+
+    is_encoder_decoder: bool = schema_utils.Boolean(
+        default=True,
+        description="",
+    )
+
+    use_cache: bool = schema_utils.Boolean(
+        default=True,
+        description="",
+    )
+
+    tokenizer_class: str = schema_utils.String(
+        default="T5Tokenizer",
+        description="",
+    )
+
+    tie_word_embeddings: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether the model's input and output word embeddings should be tied.",
+    )
+
+    pad_token_id: int = schema_utils.Integer(
+        default=0,
+        description="The ID of the token to use as padding.",
+    )
+
+    eos_token_id: int = schema_utils.Integer(
+        default=1,
+        description="The end of sequence token ID.",
+    )
+
+    decoder_start_token_id: int = schema_utils.Integer(
+        default=0,
+        description="If an encoder-decoder model starts decoding with a different token than _bos_, the id of that "
+        "token.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class XLMRoBERTaConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an XLMRoBERTa encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["xlmroberta"],
+        default="xlmroberta",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="xlm-roberta-base",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="cls_pooled",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Vocabulary size of the XLMRoBERTa model.",
+    )
+
+    pad_token_id: int = schema_utils.Integer(
+        default=1,
+        description="The ID of the token to use as padding.",
+    )
+
+    bos_token_id: int = schema_utils.Integer(
+        default=0,
+        description="The beginning of sequence token ID.",
+    )
+
+    eos_token_id: int = schema_utils.Integer(
+        default=2,
+        description="The end of sequence token ID.",
+    )
+
+    add_pooling_layer: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to add a pooling layer to the encoder.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class BERTConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an BERT encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["bert"],
+        default="bert",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="bert-base-uncased",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="cls_pooled",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=30522,
+        description="Vocabulary size of the BERT model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling BertModel or TFBertModel.",
+    )
+
+    hidden_size: int = schema_utils.PositiveInteger(
+        default=768,
+        description="Dimensionality of the encoder layers and the pooler layer.",
+    )
+
+    num_hidden_layers: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    num_attention_heads: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    intermediate_size: int = schema_utils.PositiveInteger(
+        default=3072,
+        description="Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.",
+    )
+
+    hidden_act: Union[str, Callable] = schema_utils.StringOptions(  # TODO: add support for callable
+        ["gelu", "relu", "silu", "gelu_new"],
+        default="gelu",
+        description="The non-linear activation function (function or string) in the encoder and pooler.",
+    )
+
+    hidden_dropout_prob: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    attention_probs_dropout_prob: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout ratio for the attention probabilities.",
+    )
+
+    max_position_embeddings: int = schema_utils.PositiveInteger(
+        default=512,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large just in case (e.g., 512 or 1024 or 2048).",
+    )
+
+    type_vocab_size: int = schema_utils.PositiveInteger(
+        default=2,
+        description="The vocabulary size of the token_type_ids passed when calling BertModel or TFBertModel.",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    layer_norm_eps: float = schema_utils.NonNegativeFloat(
+        default=1e-12,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    pad_token_id: int = schema_utils.Integer(
+        default=0,
+        description="The ID of the token to use as padding.",
+    )
+
+    gradient_checkpointing: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to use gradient checkpointing.",
+    )
+
+    position_embedding_type: str = schema_utils.StringOptions(
+        ["absolute", "relative_key", "relative_key_query"],
+        default="absolute",
+        description="Type of position embedding.",
+    )
+
+    classifier_dropout: float = schema_utils.FloatRange(
+        default=None,
+        min=0,
+        max=1,
+        description="The dropout ratio for the classification head.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class XLMConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an XLM encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["xlm"],
+        default="xlm",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="xlm-mlm-en-2048",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="cls_pooled",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=30145,
+        description="Vocabulary size of the BERT model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling XLMModel or TFXLMModel.",
+    )
+
+    emb_dim: int = schema_utils.PositiveInteger(
+        default=2048,
+        description="Dimensionality of the encoder layers and the pooler layer.",
+    )
+
+    n_layers: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    n_heads: int = schema_utils.PositiveInteger(
+        default=16,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    attention_dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for the attention mechanism.",
+    )
+
+    gelu_activation: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether or not to use gelu for the activations instead of relu.",
+    )
+
+    sinusoidal_embeddings: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not to use sinusoidal positional embeddings instead of absolute positional embeddings.",
+    )
+
+    causal: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not the model should behave in a causal manner. Causal models use a triangular "
+        "attention mask in order to only attend to the left-side context instead if a bidirectional "
+        "context.",
+    )
+
+    asm: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not to use an adaptive log softmax projection layer instead of a linear layer for the "
+        "prediction layer.",
+    )
+
+    n_langs: int = schema_utils.PositiveInteger(
+        default=1,
+        description="The number of languages the model handles. Set to 1 for monolingual models.",
+    )
+
+    use_lang_emb: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use language embeddings. Some models use additional language embeddings, "
+        "see the multilingual models page for information on how to use them.",
+    )
+
+    max_position_embeddings: int = schema_utils.PositiveInteger(
+        default=512,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large just in case (e.g., 512 or 1024 or 2048).",
+    )
+
+    embed_init_std: float = schema_utils.NonNegativeFloat(
+        default=2048**-0.5,
+        description="The standard deviation of the truncated_normal_initializer for initializing the embedding "
+        "matrices.",
+    )
+
+    layer_norm_eps: float = schema_utils.NonNegativeFloat(
+        default=1e-12,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    init_std: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices "
+        "except the embedding matrices.",
+    )
+
+    bos_index: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="The index of the beginning of sentence token in the vocabulary.",
+    )
+
+    eos_index: int = schema_utils.NonNegativeInteger(
+        default=1,
+        description="The index of the end of sentence token in the vocabulary.",
+    )
+
+    pad_index: int = schema_utils.NonNegativeInteger(
+        default=2,
+        description="The index of the padding token in the vocabulary.",
+    )
+
+    unk_index: int = schema_utils.NonNegativeInteger(
+        default=3, description="The index of the unknown token in the vocabulary."
+    )
+
+    mask_index: int = schema_utils.NonNegativeInteger(
+        default=5,
+        description="The index of the masking token in the vocabulary.",
+    )
+
+    is_encoder: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether or not the initialized model should be a transformer encoder or decoder as seen in "
+        "Vaswani et al.",
+    )
+
+    start_n_top: int = schema_utils.PositiveInteger(
+        default=5,
+        description="Used in the SQuAD evaluation script.",
+    )
+
+    end_n_top: int = schema_utils.PositiveInteger(
+        default=5,
+        description="Used in the SQuAD evaluation script.",
+    )
+
+    mask_token_id: int = schema_utils.Integer(
+        default=0,
+        description="Model agnostic parameter to identify masked tokens when generating text in an MLM context.",
+    )
+
+    lang_id: int = schema_utils.Integer(
+        default=0,
+        description="The ID of the language used by the model. This parameter is used when generating text in a given "
+        "language.",
+    )
+
+    pad_token_id: int = schema_utils.Integer(
+        default=2,
+        description="The ID of the token to use as padding.",
+    )
+
+    bos_token_id: int = schema_utils.Integer(
+        default=0,
+        description="The beginning of sequence token ID.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class GPTConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an GPT encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["gpt"],
+        default="gpt",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="openai-gpt",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=30522,
+        description="Vocabulary size of the GPT model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling OpenAIGPTModel or TFOpenAIGPTModel.",
+    )
+
+    n_positions: int = schema_utils.PositiveInteger(
+        default=40478,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large just in case (e.g., 512 or 1024 or 2048).",
+    )
+
+    n_ctx: int = schema_utils.PositiveInteger(
+        default=512,
+        description="Dimensionality of the causal mask (usually same as n_positions)",
+    )
+
+    n_embd: int = schema_utils.PositiveInteger(
+        default=768,
+        description="Dimensionality of the embeddings and hidden states.",
+    )
+
+    n_layer: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    n_head: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    afn: str = schema_utils.StringOptions(
+        ["gelu", "relu", "silu", "gelu_new"],
+        default="gelu_new",
+        description="The non-linear activation function (function or string) in the encoder and pooler.",
+    )
+
+    resid_pdrop: float = schema_utils.FloatRange(
+        default=0.1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    embd_pdrop: float = schema_utils.FloatRange(
+        default=0.1,
+        description="The dropout ratio for the embeddings.",
+    )
+
+    attn_pdrop: float = schema_utils.FloatRange(
+        default=0.1,
+        description="The dropout ratio for the attention.",
+    )
+
+    layer_norm_epsilon: float = schema_utils.NonNegativeFloat(
+        default=1e-5,
+        description="The epsilon to use in the layer normalization layers",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class GPT2Config(BaseEncoderConfig):
+    """This dataclass configures the schema used for an GPT2 encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["gpt2"],
+        default="gpt2",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="gpt2",
+        description="Name or path of the pretrained model.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=50257,
+        description="Vocabulary size of the GPT-2 model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling GPT2Model or TFGPT2Model.",
+    )
+
+    n_positions: int = schema_utils.PositiveInteger(
+        default=1024,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large just in case (e.g., 512 or 1024 or 2048).",
+    )
+
+    n_ctx: int = schema_utils.PositiveInteger(
+        default=1024,
+        description="Dimensionality of the causal mask (usually same as n_positions)",
+    )
+
+    n_embd: int = schema_utils.PositiveInteger(
+        default=768,
+        description="Dimensionality of the embeddings and hidden states.",
+    )
+
+    n_layer: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    n_head: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    n_inner: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Dimensionality of the inner feed-forward layers. None will set it to 4 times n_embd",
+    )
+
+    activation_function: str = schema_utils.StringOptions(
+        ["relu", "silu", "gelu", "tanh", "gelu_new"],
+        default="gelu",
+        description="Activation function, to be selected in the list ['relu', 'silu', 'gelu', 'tanh', 'gelu_new'].",
+    )
+
+    resid_pdrop: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    embd_pdrop: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout ratio for the embeddings.",
+    )
+
+    attn_pdrop: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout ratio for the attention.",
+    )
+
+    layer_norm_epsilon: float = schema_utils.NonNegativeFloat(
+        default=1e-5,
+        description="The epsilon to use in the layer normalization layers.",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    scale_attn_weights: bool = schema_utils.Boolean(
+        default=True,
+        description="Scale attention weights by dividing by sqrt(hidden_size).",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class RoBERTaConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an RoBERTa encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["roberta"],
+        default="roberta",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="roberta-base",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="cls_pooled",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Vocabulary size of the RoBERTa model.",
+    )
+
+    pad_token_id: int = schema_utils.Integer(
+        default=1,
+        description="The ID of the token to use as padding.",
+    )
+
+    bos_token_id: int = schema_utils.Integer(
+        default=0,
+        description="The beginning of sequence token ID.",
+    )
+
+    eos_token_id: int = schema_utils.Integer(
+        default=2,
+        description="The end of sequence token ID.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class TransformerXLConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an TransformerXL encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["transformer_xl"],
+        default="transformer_xl",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="transfo-xl-wt103",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=267735,
+        description="Vocabulary size of the TransfoXL model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling TransfoXLModel or TFTransfoXLModel.",
+    )
+
+    cutoffs: List[int] = schema_utils.List(
+        int,
+        default=[20000, 40000, 200000],
+        description="Cutoffs for the adaptive softmax.",
+    )
+
+    d_model: int = schema_utils.PositiveInteger(
+        default=1024,
+        description="Dimensionality of the model’s hidden states.",
+    )
+
+    d_embed: int = schema_utils.PositiveInteger(
+        default=1024,
+        description="Dimensionality of the embeddings",
+    )
+
+    n_head: int = schema_utils.PositiveInteger(
+        default=16,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    d_head: int = schema_utils.PositiveInteger(
+        default=64,
+        description="Dimensionality of the model’s heads.",
+    )
+
+    d_inner: int = schema_utils.PositiveInteger(
+        default=4096,
+        description=" Inner dimension in FF",
+    )
+
+    div_val: int = schema_utils.PositiveInteger(
+        default=4,
+        description="Divident value for adapative input and softmax.",
+    )
+
+    pre_lnorm: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not to apply LayerNorm to the input instead of the output in the blocks.",
+    )
+
+    n_layer: int = schema_utils.PositiveInteger(
+        default=18,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    mem_len: int = schema_utils.PositiveInteger(
+        default=1600,
+        description="Length of the retained previous heads.",
+    )
+
+    clamp_len: int = schema_utils.PositiveInteger(
+        default=1000,
+        description="Use the same pos embeddings after clamp_len.",
+    )
+
+    same_length: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether or not to use the same attn length for all tokens",
+    )
+
+    proj_share_all_but_first: bool = schema_utils.Boolean(
+        default=True,
+        description="True to share all but first projs, False not to share.",
+    )
+
+    attn_type: int = schema_utils.IntegerRange(
+        default=0,
+        min=0,
+        max=3,
+        description="Attention type. 0 for Transformer-XL, 1 for Shaw et al, 2 for Vaswani et al, 3 for Al Rfou et al.",
+    )
+
+    sample_softmax: int = schema_utils.Integer(
+        default=-1,
+        description="Number of samples in the sampled softmax.",
+    )
+
+    adaptive: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether or not to use adaptive softmax.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    dropatt: float = schema_utils.NonNegativeFloat(
+        default=0.0,
+        description="The dropout ratio for the attention probabilities.",
+    )
+
+    untie_r: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether ot not to untie relative position biases.",
+    )
+
+    init: str = schema_utils.String(
+        default="normal",
+        description="Parameter initializer to use.",
+    )
+
+    init_range: float = schema_utils.NonNegativeFloat(
+        default=0.01,
+        description="Parameters initialized by U(-init_range, init_range).",
+    )
+
+    proj_init_std: float = schema_utils.NonNegativeFloat(
+        default=0.01,
+        description="Parameters initialized by N(0, init_std)",
+    )
+
+    init_std: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="Parameters initialized by N(0, init_std)",
+    )
+
+    layer_norm_epsilon: float = schema_utils.NonNegativeFloat(
+        default=1e-5,
+        description="The epsilon to use in the layer normalization layers",
+    )
+
+    eos_token_id: int = schema_utils.Integer(
+        default=0,
+        description="The end of sequence token ID.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class XLNetConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an XLNet encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["xlnet"],
+        default="xlnet",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="xlnet-base-cased",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=32000,
+        description="Vocabulary size of the XLNet model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling XLNetModel or TFXLNetModel.",
+    )
+
+    d_model: int = schema_utils.PositiveInteger(
+        default=1024,
+        description="Dimensionality of the encoder layers and the pooler layer.",
+    )
+
+    n_layer: int = schema_utils.PositiveInteger(
+        default=24,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    n_head: int = schema_utils.PositiveInteger(
+        default=16,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    d_inner: int = schema_utils.PositiveInteger(
+        default=4096,
+        description="Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.",
+    )
+
+    ff_activation: str = schema_utils.StringOptions(
+        ["gelu", "relu", "silu", "gelu_new"],
+        default="gelu",
+        description="The non-linear activation function (function or string) in the encoder and pooler. If string, "
+        "'gelu', 'relu', 'silu' and 'gelu_new' are supported.",
+    )
+
+    untie_r: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether or not to untie relative position biases",
+    )
+
+    attn_type: str = schema_utils.StringOptions(
+        ["bi", "uni"],
+        default="bi",
+        description="The attention type used by the model. Set 'bi' for XLNet, 'uni' for Transformer-XL.",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    layer_norm_eps: float = schema_utils.NonNegativeFloat(
+        default=1e-12,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    mem_len: int = schema_utils.PositiveInteger(
+        default=512,
+        description="The number of tokens to cache. The key/value pairs that have already been pre-computed in a "
+        "previous forward pass won’t be re-computed. ",
+    )
+
+    reuse_len: int = schema_utils.PositiveInteger(
+        default=None,
+        description="The number of tokens in the current batch to be cached and reused in the future.",
+    )
+
+    use_mems_eval: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether or not the model should make use of the recurrent memory mechanism in evaluation mode.",
+    )
+
+    use_mems_train: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not the model should make use of the recurrent memory mechanism in train mode.",
+    )
+
+    bi_data: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not to use bidirectional input pipeline. Usually set to True during pretraining and "
+        "False during finetuning.",
+    )
+
+    clamp_len: int = schema_utils.Integer(
+        default=-1,
+        description="Clamp all relative distances larger than clamp_len. Setting this attribute to -1 means no "
+        "clamping.",
+    )
+
+    same_length: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not to use the same attention length for each token.",
+    )
+
+    summary_type: str = schema_utils.StringOptions(
+        ["last", "first", "mean", "cls_index", "attn"],
+        default="last",
+        description="Argument used when doing sequence summary. Used in the sequence classification and multiple "
+        "choice models.",
+    )
+
+    summary_use_proj: bool = schema_utils.Boolean(
+        default=True,
+        description="",
+    )
+
+    summary_activation: str = schema_utils.String(
+        default="tanh",
+        description="Argument used when doing sequence summary. Used in the sequence classification and multiple "
+        "choice models.",
+    )
+
+    summary_last_dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        description="Used in the sequence classification and multiple choice models.",
+    )
+
+    start_n_top: int = schema_utils.PositiveInteger(
+        default=5,
+        description="Used in the SQuAD evaluation script.",
+    )
+
+    end_n_top: int = schema_utils.PositiveInteger(
+        default=5,
+        description=" Used in the SQuAD evaluation script.",
+    )
+
+    pad_token_id: int = schema_utils.Integer(
+        default=5,
+        description="The ID of the token to use as padding.",
+    )
+
+    bos_token_id: int = schema_utils.Integer(
+        default=1,
+        description="The beginning of sequence token ID.",
+    )
+
+    eos_token_id: int = schema_utils.Integer(
+        default=2,
+        description="The end of sequence token ID.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class DistilBERTConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an DistilBERT encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["distilbert"],
+        default="distilbert",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="distilbert-base-uncased",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to train the model.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=30522,
+        description="Vocabulary size of the DistilBERT model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling DistilBertModel or TFDistilBertModel.",
+    )
+
+    max_position_embeddings: int = schema_utils.PositiveInteger(
+        default=512,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large just in case (e.g., 512 or 1024 or 2048).",
+    )
+
+    sinusoidal_pos_embds: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to use sinusoidal positional embeddings.",
+    )
+
+    n_layers: int = schema_utils.PositiveInteger(
+        default=6,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    n_heads: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    dim: int = schema_utils.PositiveInteger(
+        default=768,
+        description=" Dimensionality of the encoder layers and the pooler layer.",
+    )
+
+    hidden_dim: int = schema_utils.PositiveInteger(
+        default=3072,
+        description="The size of the “intermediate” (often named feed-forward) layer in the Transformer encoder.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    attention_dropout: float = schema_utils.NonNegativeFloat(
+        default=0.1,
+        description="The dropout ratio for the attention probabilities.",
+    )
+
+    activation: Union[str, Callable] = schema_utils.StringOptions(  # TODO: Add support for callable
+        ["gelu", "relu", "silu", "gelu_new"],
+        default="gelu",
+        description="The non-linear activation function (function or string) in the encoder and pooler. If string, "
+        "'gelu', 'relu', 'silu' and 'gelu_new' are supported.",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    qa_dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probabilities used in the question answering model DistilBertForQuestionAnswering.",
+    )
+
+    seq_classif_dropout: float = schema_utils.FloatRange(
+        default=0.2,
+        min=0,
+        max=1,
+        description="The dropout probabilities used in the sequence classification and the multiple choice model "
+        "DistilBertForSequenceClassification.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class CTRLConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an CTRL encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["ctrl"],
+        default="ctrl",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="ctrl",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=246534,
+        description="Vocabulary size of the CTRL model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling CTRLModel or TFCTRLModel.",
+    )
+
+    n_positions: int = schema_utils.PositiveInteger(
+        default=256,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large just in case (e.g., 512 or 1024 or 2048).",
+    )
+
+    n_ctx: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Dimensionality of the causal mask (usually same as n_positions)",
+    )
+
+    n_embd: int = schema_utils.PositiveInteger(
+        default=1280,
+        description="Dimensionality of the embeddings and hidden states.",
+    )
+
+    dff: int = schema_utils.PositiveInteger(
+        default=8192,
+        description="Dimensionality of the inner dimension of the feed forward networks (FFN).",
+    )
+
+    n_layer: int = schema_utils.PositiveInteger(
+        default=48,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    n_head: int = schema_utils.PositiveInteger(
+        default=16,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    resid_pdrop: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description=" The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    embd_pdrop: float = schema_utils.NonNegativeFloat(
+        default=0.1,
+        description="The dropout ratio for the embeddings.",
+    )
+
+    attn_pdrop: float = schema_utils.NonNegativeFloat(
+        default=0.1,
+        description="The dropout ratio for the attention.",
+    )
+
+    layer_norm_epsilon: float = schema_utils.NonNegativeFloat(
+        default=1e-6,
+        description="The epsilon to use in the layer normalization layers",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class CamemBERTConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an CamemBERT encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["camembert"],
+        default="camembert",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="jplu/camembert-base",
+        description="Name or path of the pretrained model.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="cls-pooled",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=30522,
+        description="Vocabulary size of the CamemBERT model.",
+    )
+
+    hidden_size: int = schema_utils.PositiveInteger(
+        default=768,
+        description="Dimensionality of the encoder layers and the pooler layer.",
+    )
+
+    num_hidden_layers: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    num_attention_heads: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    intermediate_size: int = schema_utils.PositiveInteger(
+        default=3072,
+        description="Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.",
+    )
+
+    hidden_act: Union[str, Callable] = schema_utils.StringOptions(  # TODO: add support for callable
+        ["gelu", "relu", "silu", "gelu_new"],
+        default="gelu",
+        description="The non-linear activation function (function or string) in the encoder and pooler.",
+    )
+
+    hidden_dropout_prob: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    attention_probs_dropout_prob: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout ratio for the attention probabilities.",
+    )
+
+    max_position_embeddings: int = schema_utils.PositiveInteger(
+        default=512,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large just in case (e.g., 512 or 1024 or 2048).",
+    )
+
+    type_vocab_size: int = schema_utils.PositiveInteger(
+        default=2,
+        description="The vocabulary size of the token_type_ids passed when calling BertModel or TFBertModel.",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    layer_norm_eps: float = schema_utils.NonNegativeFloat(
+        default=1e-12,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    pad_token_id: int = schema_utils.Integer(
+        default=0,
+        description="The ID of the token to use as padding.",
+    )
+
+    gradient_checkpointing: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to use gradient checkpointing.",
+    )
+
+    position_embedding_type: str = schema_utils.StringOptions(
+        ["absolute", "relative_key", "relative_key_query"],
+        default="absolute",
+        description="Type of position embedding.",
+    )
+
+    classifier_dropout: float = schema_utils.FloatRange(
+        default=None,
+        min=0,
+        max=1,
+        description="The dropout ratio for the classification head.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class T5Config(BaseEncoderConfig):
+    """This dataclass configures the schema used for an T5 encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["t5"],
+        default="t5",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="t5-small",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=32128,
+        description="Vocabulary size of the T5 model. Defines the number of different tokens that can be represented "
+        "by the inputs_ids passed when calling T5Model or TFT5Model.",
+    )
+
+    d_model: int = schema_utils.PositiveInteger(
+        default=512,
+        description="Size of the encoder layers and the pooler layer.",
+    )
+
+    d_kv: int = schema_utils.PositiveInteger(
+        default=64,
+        description="Size of the key, query, value projections per attention head. d_kv has to be equal to d_model // "
+        "num_heads.",
+    )
+
+    d_ff: int = schema_utils.PositiveInteger(
+        default=2048,
+        description="Size of the intermediate feed forward layer in each T5Block.",
+    )
+
+    num_layers: int = schema_utils.PositiveInteger(
+        default=6,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    num_decoder_layers: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of hidden layers in the Transformer decoder. Will use the same value as num_layers if not "
+        "set.",
+    )
+
+    num_heads: int = schema_utils.PositiveInteger(
+        default=8,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    relative_attention_num_buckets: int = schema_utils.PositiveInteger(
+        default=32,
+        description="The number of buckets to use for each attention layer.",
+    )
+
+    dropout_rate: float = schema_utils.FloatRange(
+        default=0.1,
+        description="The ratio for all dropout layers.",
+    )
+
+    layer_norm_eps: float = schema_utils.NonNegativeFloat(
+        default=1e-6,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    initializer_factor: float = schema_utils.NonNegativeFloat(
+        default=1,
+        description="A factor for initializing all weight matrices (should be kept to 1, used internally for "
+        "initialization testing).",
+    )
+
+    feed_forward_proj: str = schema_utils.StringOptions(
+        ["relu", "gated-gelu"],
+        default="relu",
+        description="Type of feed forward layer to be used. Should be one of 'relu' or 'gated-gelu'. T5v1.1 uses the "
+        "'gated-gelu' feed forward projection. Original T5 uses 'relu'.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class FlauBERTConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an FlauBERT encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["flaubert"],
+        default="flaubert",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="t5-small",
+        description="flaubert/flaubert_small_cased",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=30145,
+        description="Vocabulary size of the FlauBERT model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling FlaubertModel or TFFlaubertModel.",
+    )
+
+    pre_norm: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to apply the layer normalization before or after the feed forward layer following the "
+        "attention in each layer (Vaswani et al., Tensor2Tensor for Neural Machine Translation. 2018)",
+    )
+
+    layerdrop: float = schema_utils.FloatRange(
+        default=0.0,
+        min=0,
+        max=1,
+        description="Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand "
+        "with Structured Dropout. ICLR 2020)",
+    )
+
+    emb_dim: int = schema_utils.PositiveInteger(
+        default=2048,
+        description="Dimensionality of the encoder layers and the pooler layer.",
+    )
+
+    n_layer: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    n_head: int = schema_utils.PositiveInteger(
+        default=16,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    attention_dropout: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for the attention mechanism",
+    )
+
+    gelu_activation: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether or not to use a gelu activation instead of relu.",
+    )
+
+    sinusoidal_embeddings: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not to use sinusoidal positional embeddings instead of absolute positional embeddings.",
+    )
+
+    causal: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not the model should behave in a causal manner. Causal models use a triangular "
+        "attention mask in order to only attend to the left-side context instead if a bidirectional "
+        "context.",
+    )
+
+    asm: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether or not to use an adaptive log softmax projection layer instead of a linear layer for the "
+        "prediction layer.",
+    )
+
+    n_langs: int = schema_utils.PositiveInteger(
+        default=1,
+        description="The number of languages the model handles. Set to 1 for monolingual models.",
+    )
+
+    use_lang_emb: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use language embeddings. Some models use additional language embeddings, "
+        "see the multilingual models page for information on how to use them.",
+    )
+
+    max_position_embeddings: int = schema_utils.PositiveInteger(
+        default=512,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large just in case (e.g., 512 or 1024 or 2048).",
+    )
+
+    embed_init_std: float = schema_utils.NonNegativeFloat(
+        default=2048**-0.5,
+        description="The standard deviation of the truncated_normal_initializer for initializing the embedding "
+        "matrices.",
+    )
+
+    init_std: int = schema_utils.PositiveInteger(
+        default=50257,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices "
+        "except the embedding matrices.",
+    )
+
+    layer_norm_eps: float = schema_utils.NonNegativeFloat(
+        default=1e-12,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    bos_index: int = schema_utils.NonNegativeInteger(
+        default=0,
+        description="The index of the beginning of sentence token in the vocabulary.",
+    )
+
+    eos_index: int = schema_utils.NonNegativeInteger(
+        default=1,
+        description="The index of the end of sentence token in the vocabulary.",
+    )
+
+    pad_index: int = schema_utils.NonNegativeInteger(
+        default=2,
+        description="The index of the padding token in the vocabulary.",
+    )
+
+    unk_index: int = schema_utils.NonNegativeInteger(
+        default=3,
+        description="The index of the unknown token in the vocabulary.",
+    )
+
+    mask_index: int = schema_utils.NonNegativeInteger(
+        default=5,
+        description="The index of the masking token in the vocabulary.",
+    )
+
+    is_encoder: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether or not the initialized model should be a transformer encoder or decoder as seen in "
+        "Vaswani et al.",
+    )
+
+    mask_token_id: int = schema_utils.Integer(
+        default=0,
+        description="Model agnostic parameter to identify masked tokens when generating text in an MLM context.",
+    )
+
+    lang_id: int = schema_utils.Integer(
+        default=1,
+        description="The ID of the language used by the model. This parameter is used when generating text in a given "
+        "language.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class ELECTRAConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an ELECTRA encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["electra"],
+        default="electra",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="google/electra-small-discriminator",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=30522,
+        description="Vocabulary size of the ELECTRA model. Defines the number of different tokens that can be "
+        "represented by the inputs_ids passed when calling ElectraModel or TFElectraModel.",
+    )
+
+    embedding_size: int = schema_utils.PositiveInteger(
+        default=128,
+        description="Dimensionality of the encoder layers and the pooler layer.",
+    )
+
+    hidden_size: int = schema_utils.PositiveInteger(
+        default=256,
+        description="Dimensionality of the encoder layers and the pooler layer.",
+    )
+
+    num_hidden_layers: int = schema_utils.PositiveInteger(
+        default=12,
+        description="Number of hidden layers in the Transformer encoder.",
+    )
+
+    num_attention_heads: int = schema_utils.PositiveInteger(
+        default=4,
+        description="Number of attention heads for each attention layer in the Transformer encoder.",
+    )
+
+    intermediate_size: int = schema_utils.PositiveInteger(
+        default=1024,
+        description="Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.",
+    )
+
+    hidden_act: Union[str, Callable] = schema_utils.StringOptions(  # TODO: add support for callable
+        ["gelu", "relu", "silu", "gelu_new"],
+        default="gelu",
+        description="The non-linear activation function (function or string) in the encoder and pooler.",
+    )
+
+    hidden_dropout_prob: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.",
+    )
+
+    attention_probs_dropout_prob: float = schema_utils.FloatRange(
+        default=0.1,
+        min=0,
+        max=1,
+        description="The dropout ratio for the attention probabilities.",
+    )
+
+    max_position_embeddings: int = schema_utils.PositiveInteger(
+        default=512,
+        description="The maximum sequence length that this model might ever be used with. Typically set this to "
+        "something large just in case (e.g., 512 or 1024 or 2048).",
+    )
+
+    type_vocab_size: int = schema_utils.PositiveInteger(
+        default=2,
+        description="The vocabulary size of the token_type_ids passed when calling ElectraModel or TFElectraModel.",
+    )
+
+    initializer_range: float = schema_utils.NonNegativeFloat(
+        default=0.02,
+        description="The standard deviation of the truncated_normal_initializer for initializing all weight matrices.",
+    )
+
+    layer_norm_eps: float = schema_utils.NonNegativeFloat(
+        default=1e-12,
+        description="The epsilon used by the layer normalization layers.",
+    )
+
+    position_embedding_type: str = schema_utils.StringOptions(
+        ["absolute", "relative_key", "relative_key_query"],
+        default="absolute",
+        description="Type of position embedding.",
+    )
+
+    classifier_dropout: float = schema_utils.FloatRange(
+        default=None,
+        min=0,
+        max=1,
+        description="The dropout ratio for the classification head.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class LongformerConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an Longformer encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["longformer"],
+        default="longformer",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    use_pretrained: bool = schema_utils.Boolean(
+        default=True,
+        description="Whether to use the pretrained weights for the model.",
+    )
+
+    attention_window: Union[List[int], int] = schema_utils.IntegerOrSequenceOfIntegers(
+        default=512,
+        description="Size of an attention window around each token. If an int, use the same size for all layers. To "
+        "specify a different window size for each layer, use a List[int] where len(attention_window) == "
+        "num_hidden_layers.",
+    )
+
+    sep_token_id: int = schema_utils.Integer(
+        default=2,
+        description="ID of the separator token, which is used when building a sequence from multiple sequences",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="allenai/longformer-base-4096",
+        description="Name or path of the pretrained model.",
+    )
+
+    saved_weights_in_checkpoint: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to save the weights in the checkpoint.",
+    )
+
+    reduce_output: str = schema_utils.String(
+        default="cls_pooled",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Vocabulary size of the Longformer model.",
+    )
+
+    num_tokens: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Number of tokens",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
+
+
+@dataclass
+class AutoTransformerConfig(BaseEncoderConfig):
+    """This dataclass configures the schema used for an AutoTransformer encoder."""
+
+    type: str = schema_utils.StringOptions(
+        ["auto_transformer"],
+        default="auto_transformer",
+        allow_none=False,
+        description="Type of encoder.",
+    )
+
+    pretrained_model_name_or_path: str = schema_utils.String(
+        default="xlm-roberta-base",
+        description="Name or path of the pretrained model.",
+    )
+
+    max_sequence_length: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Maximum length of the input sequence.",
+    )
+
+    reduce_output: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="The method used to reduce a sequence of tensors down to a single tensor.",
+    )
+
+    trainable: bool = schema_utils.Boolean(
+        default=False,
+        description="Whether to train the model.",
+    )
+
+    vocab: list = schema_utils.List(
+        default=None,
+        description="Vocabulary for the encoder",
+    )
+
+    vocab_size: int = schema_utils.PositiveInteger(
+        default=None,
+        description="Vocabulary size of the XLMRoBERTa model.",
+    )
+
+    pretrained_kwargs: dict = schema_utils.Dict(
+        default=None,
+        description="Additional kwargs to pass to the pretrained model.",
+    )
diff --git a/ludwig/schema/encoders/utils.py b/ludwig/schema/encoders/utils.py
new file mode 100644
index 00000000000..21ec2d3d135
--- /dev/null
+++ b/ludwig/schema/encoders/utils.py
@@ -0,0 +1,82 @@
+from dataclasses import field
+
+from marshmallow import fields, ValidationError
+
+from ludwig.constants import TYPE
+from ludwig.encoders.registry import get_encoder_classes, get_encoder_cls
+from ludwig.schema import utils as schema_utils
+
+
+def get_encoder_conds(feature_type: str):
+    """Returns a JSON schema of conditionals to validate against encoder types for specific feature types."""
+    conds = []
+    for encoder in get_encoder_classes(feature_type):
+        encoder_cls = get_encoder_cls(feature_type, encoder).get_schema_cls()
+        other_props = schema_utils.unload_jsonschema_from_marshmallow_class(encoder_cls)["properties"]
+        other_props.pop("type")
+        encoder_cond = schema_utils.create_cond(
+            {"type": encoder},
+            other_props,
+        )
+        conds.append(encoder_cond)
+    return conds
+
+
+def EncoderDataclassField(feature_type: str, default: str):
+    """Custom dataclass field that when used inside a dataclass will allow the user to specify an encoder config.
+
+    Returns: Initialized dataclass field that converts an untyped dict with params to an encoder config.
+    """
+
+    class EncoderMarshmallowField(fields.Field):
+        """Custom marshmallow field that deserializes a dict for a valid encoder config from the encoder_registry
+        and creates a corresponding `oneOf` JSON schema for external usage."""
+
+        def _deserialize(self, value, attr, data, **kwargs):
+            if value is None:
+                return None
+            if isinstance(value, dict):
+                if TYPE in value and value[TYPE] in get_encoder_classes(feature_type):
+                    enc = get_encoder_cls(feature_type, value[TYPE]).get_schema_cls()
+                    try:
+                        return enc.Schema().load(value)
+                    except (TypeError, ValidationError) as error:
+                        raise ValidationError(
+                            f"Invalid encoder params: {value}, see `{enc}` definition. Error: {error}"
+                        )
+                raise ValidationError(
+                    f"Invalid params for encoder: {value}, expect dict with at least a valid `type` attribute."
+                )
+            raise ValidationError("Field should be None or dict")
+
+        @staticmethod
+        def _jsonschema_type_mapping():
+            encoder_classes = list(get_encoder_classes(feature_type).keys())
+
+            return {
+                "type": "object",
+                "properties": {
+                    "type": {"type": "string", "enum": encoder_classes, "default": default},
+                },
+                "title": "encoder_options",
+                "allOf": get_encoder_conds(feature_type),
+                "required": ["type"],
+            }
+
+    try:
+        encoder = get_encoder_cls(feature_type, default).get_schema_cls()
+        load_default = encoder.Schema().load({"type": default})
+        dump_default = encoder.Schema().dump({"type": default})
+
+        return field(
+            metadata={
+                "marshmallow_field": EncoderMarshmallowField(
+                    allow_none=False,
+                    dump_default=dump_default,
+                    load_default=load_default,
+                )
+            },
+            default_factory=lambda: load_default,
+        )
+    except Exception as e:
+        raise ValidationError(f"Unsupported encoder type: {default}. See encoder_registry. " f"Details: {e}")
diff --git a/ludwig/schema/features/audio_feature.py b/ludwig/schema/features/audio_feature.py
index a5c17e8a444..585920792fe 100644
--- a/ludwig/schema/features/audio_feature.py
+++ b/ludwig/schema/features/audio_feature.py
@@ -1,10 +1,9 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
 from ludwig.constants import AUDIO
-from ludwig.encoders.registry import get_encoder_classes
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -15,8 +14,14 @@ class AudioInputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=AUDIO)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(AUDIO).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=AUDIO,
         default="parallel_cnn",
-        description="Encoder to use for this audio feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
diff --git a/ludwig/schema/features/bag_feature.py b/ludwig/schema/features/bag_feature.py
index b807ee6d79a..66a10037551 100644
--- a/ludwig/schema/features/bag_feature.py
+++ b/ludwig/schema/features/bag_feature.py
@@ -1,10 +1,9 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
 from ludwig.constants import BAG
-from ludwig.encoders.registry import get_encoder_classes
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -15,8 +14,14 @@ class BagInputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=BAG)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(BAG).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=BAG,
         default="embed",
-        description="Encoder to use for this bag feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
diff --git a/ludwig/schema/features/base.py b/ludwig/schema/features/base.py
index 8d9ab555233..90be6b714bb 100644
--- a/ludwig/schema/features/base.py
+++ b/ludwig/schema/features/base.py
@@ -1,7 +1,57 @@
+from typing import List
+
+from marshmallow_dataclass import dataclass
+
+from ludwig.constants import (
+    AUDIO,
+    BAG,
+    BINARY,
+    CATEGORY,
+    DATE,
+    H3,
+    IMAGE,
+    NUMBER,
+    SEQUENCE,
+    SET,
+    TEXT,
+    TIMESERIES,
+    VECTOR,
+)
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.metadata.parameter_metadata import ParameterMetadata
+
+
+@dataclass
+class BaseFeatureConfig(schema_utils.BaseMarshmallowConfig):
+    """Base class for feature configs."""
+
+    name: str = schema_utils.String(
+        allow_none=True,
+        description="Name of the feature.",
+    )
 
+    type: str = schema_utils.StringOptions(
+        allow_none=True,
+        options=[AUDIO, BAG, BINARY, CATEGORY, DATE, H3, IMAGE, NUMBER, SEQUENCE, SET, TEXT, TIMESERIES, VECTOR],
+        description="Type of the feature.",
+    )
 
-class BaseInputFeatureConfig(schema_utils.BaseMarshmallowConfig):
+    column: str = schema_utils.String(
+        allow_none=True,
+        default=None,
+        description="The column name of this feature. Defaults to name if not specified.",
+    )
+
+    proc_column: str = schema_utils.String(
+        allow_none=True,
+        default=None,
+        description="The name of the preprocessed column name of this feature. Internal only.",
+        parameter_metadata=ParameterMetadata(internal_only=True),
+    )
+
+
+@dataclass
+class BaseInputFeatureConfig(BaseFeatureConfig):
     """Base input feature config class."""
 
     tied: str = schema_utils.String(
@@ -12,7 +62,22 @@ class BaseInputFeatureConfig(schema_utils.BaseMarshmallowConfig):
     )
 
 
-class BaseOutputFeatureConfig(schema_utils.BaseMarshmallowConfig):
+@dataclass
+class BaseOutputFeatureConfig(BaseFeatureConfig):
     """Base output feature config class."""
 
-    pass
+    reduce_input: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first "
+        "dimension (second if you count the batch dimension)",
+    )
+
+    dependencies: List[str] = schema_utils.List(
+        default=[],
+        description="List of input features that this feature depends on.",
+    )
+
+    reduce_dependencies: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the dependencies of the output feature.",
+    )
diff --git a/ludwig/schema/features/binary_feature.py b/ludwig/schema/features/binary_feature.py
index 875433edd38..6440197d370 100644
--- a/ludwig/schema/features/binary_feature.py
+++ b/ludwig/schema/features/binary_feature.py
@@ -1,35 +1,66 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
-from ludwig.constants import BINARY
-from ludwig.decoders.registry import get_decoder_classes
-from ludwig.encoders.registry import get_encoder_classes
+from ludwig.constants import BINARY, BINARY_WEIGHTED_CROSS_ENTROPY
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.decoders.base import BaseDecoderConfig
+from ludwig.schema.decoders.utils import DecoderDataclassField
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
 
 @dataclass
 class BinaryInputFeatureConfig(BaseInputFeatureConfig):
-    """BinaryInputFeature is a dataclass that configures the parameters used for a binary input feature."""
+    """BinaryInputFeatureConfig is a dataclass that configures the parameters used for a binary input feature."""
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=BINARY)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(BINARY).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=BINARY,
         default="passthrough",
-        description="Encoder to use for this binary feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
 
 
 @dataclass
 class BinaryOutputFeatureConfig(BaseOutputFeatureConfig):
-    """BinaryOutputFeature is a dataclass that configures the parameters used for a binary output feature."""
+    """BinaryOutputFeatureConfig is a dataclass that configures the parameters used for a binary output feature."""
 
-    decoder: Optional[str] = schema_utils.StringOptions(
-        list(get_decoder_classes(BINARY).keys()),
+    loss: dict = schema_utils.Dict(  # TODO: Create schema for loss
+        default={
+            "type": BINARY_WEIGHTED_CROSS_ENTROPY,
+            "robust_lambda": 0,
+            "confidence_penalty": 0,
+            "positive_class_weight": None,
+            "weight": 1,
+        },
+        description="A dictionary containing a loss type and its hyper-parameters.",
+    )
+
+    decoder: BaseDecoderConfig = DecoderDataclassField(
+        feature_type=BINARY,
         default="regressor",
-        allow_none=True,
-        description="Decoder to use for this binary feature.",
+    )
+
+    reduce_input: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first "
+        "dimension (second if you count the batch dimension)",
+    )
+
+    dependencies: list = schema_utils.List(
+        default=[],
+        description="List of input features that this feature depends on.",
+    )
+
+    reduce_dependencies: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the dependencies of the output feature.",
     )
diff --git a/ludwig/schema/features/category_feature.py b/ludwig/schema/features/category_feature.py
index 9b3bdd6dddf..557f68cd765 100644
--- a/ludwig/schema/features/category_feature.py
+++ b/ludwig/schema/features/category_feature.py
@@ -1,35 +1,76 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
-from ludwig.constants import CATEGORY
-from ludwig.decoders.registry import get_decoder_classes
-from ludwig.encoders.registry import get_encoder_classes
+from ludwig.constants import CATEGORY, SOFTMAX_CROSS_ENTROPY
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.decoders.base import BaseDecoderConfig
+from ludwig.schema.decoders.utils import DecoderDataclassField
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
 
 @dataclass
 class CategoryInputFeatureConfig(BaseInputFeatureConfig):
-    """CategoryInputFeature is a dataclass that configures the parameters used for a category input feature."""
+    """CategoryInputFeatureConfig is a dataclass that configures the parameters used for a category input
+    feature."""
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=CATEGORY)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(CATEGORY).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=CATEGORY,
         default="dense",
-        description="Encoder to use for this category feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
 
 
 @dataclass
 class CategoryOutputFeatureConfig(BaseOutputFeatureConfig):
-    """CategoryOutputFeature is a dataclass that configures the parameters used for a category output feature."""
+    """CategoryOutputFeatureConfig is a dataclass that configures the parameters used for a category output
+    feature."""
 
-    decoder: Optional[str] = schema_utils.StringOptions(
-        list(get_decoder_classes(CATEGORY).keys()),
+    loss: dict = schema_utils.Dict(  # TODO: Schema for loss
+        default={
+            "type": SOFTMAX_CROSS_ENTROPY,
+            "class_weights": 1,
+            "robust_lambda": 0,
+            "confidence_penalty": 0,
+            "class_similarities_temperature": 0,
+            "weight": 1,
+        },
+        description="A dictionary containing a loss type and its hyper-parameters.",
+    )
+
+    decoder: BaseDecoderConfig = DecoderDataclassField(
+        feature_type=CATEGORY,
         default="classifier",
-        allow_none=True,
-        description="Decoder to use for this category feature.",
+    )
+
+    reduce_input: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first "
+        "dimension (second if you count the batch dimension)",
+    )
+
+    dependencies: list = schema_utils.List(
+        default=[],
+        description="List of input features that this feature depends on.",
+    )
+
+    reduce_dependencies: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the dependencies of the output feature.",
+    )
+
+    top_k: int = schema_utils.NonNegativeInteger(
+        default=3,
+        description="Determines the parameter k, the number of categories to consider when computing the top_k "
+        "measure. It computes accuracy but considering as a match if the true category appears in the "
+        "first k predicted categories ranked by decoder's confidence.",
     )
diff --git a/ludwig/schema/features/date_feature.py b/ludwig/schema/features/date_feature.py
index 8cecd68102c..743c0f4bb19 100644
--- a/ludwig/schema/features/date_feature.py
+++ b/ludwig/schema/features/date_feature.py
@@ -1,10 +1,9 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
 from ludwig.constants import DATE
-from ludwig.encoders.registry import get_encoder_classes
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -15,8 +14,14 @@ class DateInputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=DATE)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(DATE).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=DATE,
         default="embed",
-        description="Encoder to use for this date feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
diff --git a/ludwig/schema/features/h3_feature.py b/ludwig/schema/features/h3_feature.py
index 32b04bcda56..bfe3c63a169 100644
--- a/ludwig/schema/features/h3_feature.py
+++ b/ludwig/schema/features/h3_feature.py
@@ -1,10 +1,9 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
 from ludwig.constants import H3
-from ludwig.encoders.registry import get_encoder_classes
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -15,8 +14,14 @@ class H3InputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=H3)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(H3).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=H3,
         default="embed",
-        description="Encoder to use for this h3 feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
diff --git a/ludwig/schema/features/image_feature.py b/ludwig/schema/features/image_feature.py
index 489036e45da..8c0144eef43 100644
--- a/ludwig/schema/features/image_feature.py
+++ b/ludwig/schema/features/image_feature.py
@@ -1,10 +1,9 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
 from ludwig.constants import IMAGE
-from ludwig.encoders.registry import get_encoder_classes
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -15,8 +14,14 @@ class ImageInputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=IMAGE)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(IMAGE).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=IMAGE,
         default="stacked_cnn",
-        description="Encoder to use for this image feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
diff --git a/ludwig/schema/features/number_feature.py b/ludwig/schema/features/number_feature.py
index 6df975f577f..16bfcd86eba 100644
--- a/ludwig/schema/features/number_feature.py
+++ b/ludwig/schema/features/number_feature.py
@@ -1,34 +1,57 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
-from ludwig.constants import NUMBER
-from ludwig.decoders.registry import get_decoder_classes
-from ludwig.encoders.registry import get_encoder_classes
+from ludwig.constants import MEAN_SQUARED_ERROR, NUMBER
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.decoders.base import BaseDecoderConfig
+from ludwig.schema.decoders.utils import DecoderDataclassField
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
 
 @dataclass
 class NumberInputFeatureConfig(BaseInputFeatureConfig):
-    """NumberInputFeature is a dataclass that configures the parameters used for a number input feature."""
+    """NumberInputFeatureConfig is a dataclass that configures the parameters used for a number input feature."""
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=NUMBER)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(NUMBER).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type="number",
         default="passthrough",
-        description="Encoder to use for this number feature.",
     )
 
 
 @dataclass
 class NumberOutputFeatureConfig(BaseOutputFeatureConfig):
+    """NumberOutputFeatureConfig is a dataclass that configures the parameters used for a category output
+    feature."""
+
+    loss: dict = schema_utils.Dict(
+        default={
+            "type": MEAN_SQUARED_ERROR,
+            "weight": 1,
+        },
+        description="A dictionary containing a loss type and its hyper-parameters.",
+    )
 
-    decoder: Optional[str] = schema_utils.StringOptions(
-        list(get_decoder_classes(NUMBER).keys()),
+    decoder: BaseDecoderConfig = DecoderDataclassField(
+        feature_type=NUMBER,
         default="regressor",
-        allow_none=True,
-        description="Decoder to use for this number feature.",
+    )
+
+    reduce_input: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first "
+        "dimension (second if you count the batch dimension)",
+    )
+
+    dependencies: list = schema_utils.List(
+        default=[],
+        description="List of input features that this feature depends on.",
+    )
+
+    reduce_dependencies: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the dependencies of the output feature.",
     )
diff --git a/ludwig/schema/features/sequence_feature.py b/ludwig/schema/features/sequence_feature.py
index 7476af77473..6e272da9189 100644
--- a/ludwig/schema/features/sequence_feature.py
+++ b/ludwig/schema/features/sequence_feature.py
@@ -1,11 +1,11 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
-from ludwig.constants import SEQUENCE
-from ludwig.decoders.registry import get_decoder_classes
-from ludwig.encoders.registry import get_encoder_classes
+from ludwig.constants import SEQUENCE, SEQUENCE_SOFTMAX_CROSS_ENTROPY
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.decoders.base import BaseDecoderConfig
+from ludwig.schema.decoders.utils import DecoderDataclassField
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -17,10 +17,9 @@ class SequenceInputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=SEQUENCE)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(SEQUENCE).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=SEQUENCE,
         default="embed",
-        description="Encoder to use for this sequence feature.",
     )
 
 
@@ -29,9 +28,36 @@ class SequenceOutputFeatureConfig(BaseOutputFeatureConfig):
     """SequenceOutputFeatureConfig is a dataclass that configures the parameters used for a sequence output
     feature."""
 
-    decoder: Optional[str] = schema_utils.StringOptions(
-        list(get_decoder_classes(SEQUENCE).keys()),
+    loss: dict = schema_utils.Dict(  # TODO: Schema for loss
+        default={
+            "type": SEQUENCE_SOFTMAX_CROSS_ENTROPY,
+            "class_weights": 1,
+            "robust_lambda": 0,
+            "confidence_penalty": 0,
+            "class_similarities_temperature": 0,
+            "weight": 1,
+            "unique": False,
+        },
+        description="A dictionary containing a loss type and its hyper-parameters.",
+    )
+
+    decoder: BaseDecoderConfig = DecoderDataclassField(
+        feature_type=SEQUENCE,
         default="generator",
-        allow_none=True,
-        description="Decoder to use for this sequence feature.",
+    )
+
+    reduce_input: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first "
+        "dimension (second if you count the batch dimension)",
+    )
+
+    dependencies: list = schema_utils.List(
+        default=[],
+        description="List of input features that this feature depends on.",
+    )
+
+    reduce_dependencies: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the dependencies of the output feature.",
     )
diff --git a/ludwig/schema/features/set_feature.py b/ludwig/schema/features/set_feature.py
index b438887d344..8c4504f1fe9 100644
--- a/ludwig/schema/features/set_feature.py
+++ b/ludwig/schema/features/set_feature.py
@@ -1,11 +1,11 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
-from ludwig.constants import SET
-from ludwig.decoders.registry import get_decoder_classes
-from ludwig.encoders.registry import get_encoder_classes
+from ludwig.constants import SET, SIGMOID_CROSS_ENTROPY
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.decoders.base import BaseDecoderConfig
+from ludwig.schema.decoders.utils import DecoderDataclassField
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -16,10 +16,16 @@ class SetInputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=SET)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(SET).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=SET,
         default="embed",
-        description="Encoder to use for this set feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
 
 
@@ -27,9 +33,32 @@ class SetInputFeatureConfig(BaseInputFeatureConfig):
 class SetOutputFeatureConfig(BaseOutputFeatureConfig):
     """SetOutputFeatureConfig is a dataclass that configures the parameters used for a set output feature."""
 
-    decoder: Optional[str] = schema_utils.StringOptions(
-        list(get_decoder_classes(SET).keys()),
+    loss: dict = schema_utils.Dict(
+        default={
+            "type": SIGMOID_CROSS_ENTROPY,
+            "class_weights": None,
+            "weight": 1,
+        },
+        description="A dictionary containing a loss type and its hyper-parameters.",
+    )
+
+    decoder: BaseDecoderConfig = DecoderDataclassField(
+        feature_type=SET,
         default="classifier",
-        allow_none=True,
-        description="Decoder to use for this set feature.",
+    )
+
+    reduce_input: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first "
+        "dimension (second if you count the batch dimension)",
+    )
+
+    dependencies: list = schema_utils.List(
+        default=[],
+        description="List of input features that this feature depends on.",
+    )
+
+    reduce_dependencies: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the dependencies of the output feature.",
     )
diff --git a/ludwig/schema/features/text_feature.py b/ludwig/schema/features/text_feature.py
index 615ce3ebb3c..efe49ea9d04 100644
--- a/ludwig/schema/features/text_feature.py
+++ b/ludwig/schema/features/text_feature.py
@@ -1,11 +1,11 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
-from ludwig.constants import TEXT
-from ludwig.decoders.registry import get_decoder_classes
-from ludwig.encoders.registry import get_encoder_classes
+from ludwig.constants import SEQUENCE_SOFTMAX_CROSS_ENTROPY, TEXT
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.decoders.base import BaseDecoderConfig
+from ludwig.schema.decoders.utils import DecoderDataclassField
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -16,10 +16,9 @@ class TextInputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=TEXT)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(TEXT).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=TEXT,
         default="parallel_cnn",
-        description="Encoder to use for this text feature.",
     )
 
 
@@ -27,8 +26,36 @@ class TextInputFeatureConfig(BaseInputFeatureConfig):
 class TextOutputFeatureConfig(BaseOutputFeatureConfig):
     """TextOutputFeatureConfig is a dataclass that configures the parameters used for a text output feature."""
 
-    decoder: Optional[str] = schema_utils.StringOptions(
-        list(get_decoder_classes(TEXT).keys()),
+    loss: dict = schema_utils.Dict(
+        default={
+            "type": SEQUENCE_SOFTMAX_CROSS_ENTROPY,
+            "class_weights": 1,
+            "robust_lambda": 0,
+            "confidence_penalty": 0,
+            "class_similarities_temperature": 0,
+            "weight": 1,
+            "unique": False,
+        },
+        description="A dictionary containing a loss type and its hyper-parameters.",
+    )
+
+    decoder: BaseDecoderConfig = DecoderDataclassField(
+        feature_type=TEXT,
         default="generator",
-        description="Decoder to use for this text output feature.",
+    )
+
+    reduce_input: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first "
+        "dimension (second if you count the batch dimension)",
+    )
+
+    dependencies: list = schema_utils.List(
+        default=[],
+        description="List of input features that this feature depends on.",
+    )
+
+    reduce_dependencies: str = schema_utils.ReductionOptions(
+        default="sum",
+        description="How to reduce the dependencies of the output feature.",
     )
diff --git a/ludwig/schema/features/timeseries_feature.py b/ludwig/schema/features/timeseries_feature.py
index 3a4be08aa05..e7d8518f153 100644
--- a/ludwig/schema/features/timeseries_feature.py
+++ b/ludwig/schema/features/timeseries_feature.py
@@ -1,10 +1,9 @@
-from typing import Optional
-
 from marshmallow_dataclass import dataclass
 
 from ludwig.constants import TIMESERIES
-from ludwig.encoders.registry import get_encoder_classes
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -16,8 +15,14 @@ class TimeseriesInputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=TIMESERIES)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(TIMESERIES).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=TIMESERIES,
         default="parallel_cnn",
-        description="Encoder to use for this timeseries feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
diff --git a/ludwig/schema/features/utils.py b/ludwig/schema/features/utils.py
index f2a14a35b70..469e569f5da 100644
--- a/ludwig/schema/features/utils.py
+++ b/ludwig/schema/features/utils.py
@@ -1,5 +1,3 @@
-from ludwig.decoders.registry import get_decoder_classes
-from ludwig.encoders.registry import get_encoder_classes
 from ludwig.schema import utils as schema_utils
 from ludwig.utils.registry import Registry
 
@@ -23,38 +21,6 @@ def wrap(cls):
     return wrap
 
 
-def update_encoders(feature_props, feature_type):
-    """This function updates the list of encoders acquired from the registry with any custom encoders that are not
-    registered before schema validation.
-
-    Args:
-        feature_props: Input feature properties
-        feature_type: Input feature type
-
-    Returns:
-        None
-    """
-    for key in get_encoder_classes(feature_type):
-        if key not in feature_props["encoder"]["enum"]:
-            feature_props["encoder"]["enum"].append(key)
-
-
-def update_decoders(feature_props, feature_type):
-    """This function updates the list of decoders acquired from the registry with any custom decoders that are not
-    registered before schema validation.
-
-    Args:
-        feature_props: Output feature properties
-        feature_type: Output feature type
-
-    Returns:
-        None
-    """
-    for key in get_decoder_classes(feature_type):
-        if key not in feature_props["decoder"]["enum"]:
-            feature_props["decoder"]["enum"].append(key)
-
-
 def get_input_feature_jsonschema():
     """This function returns a JSON schema structured to only requires a `type` key and then conditionally applies
     a corresponding input feature's field constraints.
@@ -91,7 +57,6 @@ def get_input_feature_conds():
         schema_cls = feature_cls.get_schema_cls()
         feature_schema = schema_utils.unload_jsonschema_from_marshmallow_class(schema_cls)
         feature_props = feature_schema["properties"]
-        update_encoders(feature_props, feature_type)
         feature_cond = schema_utils.create_cond({"type": feature_type}, feature_props)
         conds.append(feature_cond)
     return conds
@@ -133,7 +98,6 @@ def get_output_feature_conds():
         schema_cls = feature_cls.get_schema_cls()
         feature_schema = schema_utils.unload_jsonschema_from_marshmallow_class(schema_cls)
         feature_props = feature_schema["properties"]
-        update_decoders(feature_props, feature_type)
         feature_cond = schema_utils.create_cond({"type": feature_type}, feature_props)
         conds.append(feature_cond)
     return conds
diff --git a/ludwig/schema/features/vector_feature.py b/ludwig/schema/features/vector_feature.py
index a90b2f44c5a..c5c32b202da 100644
--- a/ludwig/schema/features/vector_feature.py
+++ b/ludwig/schema/features/vector_feature.py
@@ -1,11 +1,13 @@
-from typing import Optional
+from typing import List
 
 from marshmallow_dataclass import dataclass
 
-from ludwig.constants import VECTOR
-from ludwig.decoders.registry import get_decoder_classes
-from ludwig.encoders.registry import get_encoder_classes
+from ludwig.constants import MEAN_SQUARED_ERROR, VECTOR
 from ludwig.schema import utils as schema_utils
+from ludwig.schema.decoders.base import BaseDecoderConfig
+from ludwig.schema.decoders.utils import DecoderDataclassField
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
 from ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig
 from ludwig.schema.preprocessing import BasePreprocessingConfig, PreprocessingDataclassField
 
@@ -16,10 +18,16 @@ class VectorInputFeatureConfig(BaseInputFeatureConfig):
 
     preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=VECTOR)
 
-    encoder: Optional[str] = schema_utils.StringOptions(
-        list(get_encoder_classes(VECTOR).keys()),
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        feature_type=VECTOR,
         default="dense",
-        description="Encoder to use for this vector feature.",
+    )
+
+    tied: str = schema_utils.String(
+        default=None,
+        allow_none=True,
+        description="Name of input feature to tie the weights of the encoder with.  It needs to be the name of a "
+        "feature of the same type and with the same encoder parameters.",
     )
 
 
@@ -27,8 +35,31 @@ class VectorInputFeatureConfig(BaseInputFeatureConfig):
 class VectorOutputFeatureConfig(BaseOutputFeatureConfig):
     """VectorOutputFeatureConfig is a dataclass that configures the parameters used for a vector output feature."""
 
-    decoder: Optional[str] = schema_utils.StringOptions(
-        list(get_decoder_classes(VECTOR).keys()),
+    reduce_input: str = schema_utils.ReductionOptions(
+        default=None,
+        description="How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first "
+        "dimension (second if you count the batch dimension)",
+    )
+
+    reduce_dependencies: str = schema_utils.ReductionOptions(
+        default=None,
+        description="How to reduce the dependencies of the output feature.",
+    )
+
+    loss: dict = schema_utils.Dict(
+        default={
+            "type": MEAN_SQUARED_ERROR,
+            "weight": 1,
+        },
+        description="A dictionary containing a loss type and its hyper-parameters.",
+    )
+
+    decoder: BaseDecoderConfig = DecoderDataclassField(
+        feature_type=VECTOR,
         default="projector",
-        description="Decoder to use for this vector feature.",
+    )
+
+    dependencies: List = schema_utils.List(
+        default=[],
+        description="List of input features that this feature depends on.",
     )
diff --git a/ludwig/schema/metadata/preprocessing_metadata.py b/ludwig/schema/metadata/preprocessing_metadata.py
index 89f7ed89b1f..82b8f3adbde 100644
--- a/ludwig/schema/metadata/preprocessing_metadata.py
+++ b/ludwig/schema/metadata/preprocessing_metadata.py
@@ -1,7 +1,7 @@
 # flake8: noqa
 # fmt: off
 
-from ludwig.schema.metadata.parameter_metadata import ExpectedImpact, ParameterMetadata
+from ludwig.schema.metadata.parameter_metadata import ParameterMetadata
 
 PREPROCESSING_METADATA = (
     {'computed_fill_value': ParameterMetadata(internal_only=True),
diff --git a/ludwig/schema/optimizers.py b/ludwig/schema/optimizers.py
index 033fb7a3131..4ac5d8a1d62 100644
--- a/ludwig/schema/optimizers.py
+++ b/ludwig/schema/optimizers.py
@@ -309,7 +309,8 @@ def _deserialize(self, value, attr, data, **kwargs):
                 )
             raise ValidationError("Field should be None or dict")
 
-        def _jsonschema_type_mapping(self):
+        @staticmethod
+        def _jsonschema_type_mapping():
             # Note that this uses the same conditional pattern as combiners:
             return {
                 "type": "object",
@@ -326,7 +327,8 @@ def _jsonschema_type_mapping(self):
         raise ValidationError(f"Invalid default: `{default}`")
     try:
         opt = optimizer_registry[default["type"].lower()][1]
-        load_default = opt.Schema().load(default)
+        load_default = opt.Schema()
+        load_default = load_default.load(default)
         dump_default = opt.Schema().dump(default)
 
         return field(
diff --git a/ludwig/schema/preprocessing.py b/ludwig/schema/preprocessing.py
index 0584c0210c9..e60f36a6140 100644
--- a/ludwig/schema/preprocessing.py
+++ b/ludwig/schema/preprocessing.py
@@ -94,7 +94,7 @@ class TextPreprocessingConfig(schema_utils.BaseMarshmallowConfig):
     )
 
     padding_symbol: str = schema_utils.String(
-        default="<PAD>",
+        default=strings_utils.PADDING_SYMBOL,
         allow_none=False,
         description="The string used as the padding symbol for sequence features. Ignored for features using "
         "huggingface encoders, which have their own vocabulary.",
@@ -192,20 +192,20 @@ class BinaryPreprocessingConfig(schema_utils.BaseMarshmallowConfig):
         allow_none=True,
         description="The value to replace missing values with in case the missing_value_strategy is fill_with_const",
         field_options=[
-            schema_utils.FloatRange(default=None, allow_none=True, min=0, max=1),
-            schema_utils.StringOptions(options=strings_utils.all_bool_strs(), default=None),
+            schema_utils.FloatRange(allow_none=False, default=0, min=0, max=1),
+            schema_utils.StringOptions(allow_none=True, options=strings_utils.all_bool_strs()),
         ],
     )
 
     computed_fill_value: Union[int, float, str] = schema_utils.OneOfOptionsField(
         default=None,
-        allow_none=False,
+        allow_none=True,
         description="The internally computed fill value to replace missing values with in case the "
         "missing_value_strategy is fill_with_mode or fill_with_mean",
         parameter_metadata=PREPROCESSING_METADATA["computed_fill_value"],
         field_options=[
-            schema_utils.FloatRange(default=None, allow_none=True, min=0, max=1),
-            schema_utils.StringOptions(options=strings_utils.all_bool_strs(), default=None),
+            schema_utils.FloatRange(allow_none=False, default=0, min=0, max=1),
+            schema_utils.StringOptions(allow_none=True, options=strings_utils.all_bool_strs()),
         ],
     )
 
@@ -335,7 +335,7 @@ class SequencePreprocessingConfig(schema_utils.BaseMarshmallowConfig):
     )
 
     padding_symbol: str = schema_utils.String(
-        default="<PAD>",
+        default=strings_utils.PADDING_SYMBOL,
         allow_none=False,
         description="The string used as a padding symbol. This special token is mapped to the integer ID 0 in the "
         "vocabulary.",
@@ -801,6 +801,14 @@ def _deserialize(self, value, attr, data, **kwargs):
                 )
             raise ValidationError("Field should be None or dict")
 
+        def _serialize(self, value, attr, data, **kwargs):
+            if value is None:
+                return None
+            if feature_type in preprocessing_registry:
+                pre = preprocessing_registry[feature_type]
+                return pre.Schema().dump(value)
+            raise ValidationError(f"Invalid preprocessing feature type {feature_type}")
+
         @staticmethod
         def _jsonschema_type_mapping():
             preprocessor_cls = preprocessing_registry[feature_type]
diff --git a/ludwig/schema/utils.py b/ludwig/schema/utils.py
index a11cfa10e61..e1c14d1f4ef 100644
--- a/ludwig/schema/utils.py
+++ b/ludwig/schema/utils.py
@@ -1,5 +1,5 @@
 from dataclasses import field
-from typing import Any as TAny
+from typing import Any
 from typing import Dict as TDict
 from typing import List as TList
 from typing import Tuple, Type, Union
@@ -13,14 +13,16 @@
 from ludwig.utils.torch_utils import activations, initializer_registry
 
 
-def load_config(cls: Type["BaseMarshmallowConfig"], **kwargs):  # noqa 0821
+def load_config(cls: Type["BaseMarshmallowConfig"], **kwargs) -> "BaseMarshmallowConfig":  # noqa 0821
     """Takes a marshmallow class and instantiates it with the given keyword args as parameters."""
     assert_is_a_marshmallow_class(cls)
     schema = cls.Schema()
     return schema.load(kwargs)
 
 
-def load_trainer_with_kwargs(model_type: str, kwargs):  # noqa: F821
+def load_trainer_with_kwargs(
+    model_type: str, kwargs: dict
+) -> Tuple["BaseMarshmallowConfig", TDict[str, Any]]:  # noqa: F821
     """Special case of `load_config_with_kwargs` for the trainer schemas.
 
     In particular, it chooses the correct default type for an incoming config (if it doesn't have one already), but
@@ -43,8 +45,11 @@ def default_type_for_trainer_schema(cls):
 
 def load_config_with_kwargs(
     cls: Type["BaseMarshmallowConfig"], kwargs_overrides
-) -> "BaseMarshmallowConfig":  # noqa 0821
-    """Instatiates an instance of the marshmallow class and kwargs overrides instantiantes the schema."""
+) -> Tuple["BaseMarshmallowConfig", TDict[str, Any]]:  # noqa 0821
+    """Instatiates an instance of the marshmallow class and kwargs overrides instantiantes the schema.
+
+    Returns a tuple of config, and a dictionary of any keys in kwargs_overrides which are no present in config.
+    """
     assert_is_a_marshmallow_class(cls)
     schema = cls.Schema()
     fields = schema.fields.keys()
@@ -411,6 +416,143 @@ def FloatRange(
     )
 
 
+def IntegerOrSequenceOfIntegers(
+    default: Union[None, int, Tuple[int, ...], TList[int]] = None,
+    default_integer: int = None,
+    default_sequence: Union[TList[int], Tuple[int, ...]] = None,
+    allow_none=False,
+    non_negative: bool = True,
+    description="",
+):
+    """Returns a dataclass field with marshmallow metadata enforcing numeric inputs or a tuple of numeric
+    inputs."""
+
+    class IntegerOrIntegerSequenceField(fields.Field):
+        def _deserialize(self, value, attr, data, **kwargs):
+            if isinstance(value, int):
+                if non_negative:
+                    if value < 0:
+                        raise ValidationError("Value must be positive.")
+                return value
+            if isinstance(value, (tuple, list)):
+                if non_negative:
+                    for v in value:
+                        if v < 0:
+                            raise ValidationError("Values must be positive.")
+                return value
+            raise ValidationError("Field should be either an integer, tuple of integers, or a list of integers")
+
+        def _jsonschema_type_mapping(self):
+            numeric_option = {
+                "type": "integer",
+                "title": "integer_option",
+                "default": default_integer,
+                "description": "Set to a valid number.",
+            }
+            sequence_option = {
+                "type": "array",
+                "title": "sequence_option",
+                "items": {"type": "number"},
+                "default": default_sequence,
+                "description": "Set to a valid number.",
+            }
+
+            oneof_list = [
+                numeric_option,
+                sequence_option,
+            ]
+
+            return {"oneOf": oneof_list, "title": self.name, "description": description, "default": default}
+
+    return field(
+        metadata={
+            "marshmallow_field": IntegerOrIntegerSequenceField(
+                allow_none=allow_none, load_default=default, dump_default=default, metadata={"description": description}
+            )
+        },
+        default=default,
+    )
+
+
+def PositiveIntegerOrTupleOrStringOptions(
+    options: TList[str] = None,
+    allow_none=False,
+    default: Union[None, int, Tuple[int, ...], str] = None,
+    default_integer: Union[None, int] = None,
+    default_tuple: Union[None, Tuple[int, ...]] = None,
+    default_option: Union[None, str] = None,
+    description="",
+):
+    """Returns a dataclass field with marshmallow metadata enforcing numeric inputs, a tuple of numeric inputs, or
+    a string value."""
+
+    class IntegerTupleStringOptionsField(fields.Field):
+        def _deserialize(self, value, attr, data, **kwargs):
+            if isinstance(value, int):
+                if value < 0:
+                    raise ValidationError("Value must be positive.")
+                return value
+            if isinstance(value, tuple):
+                for v in value:
+                    if v < 0:
+                        raise ValidationError("Values must be positive.")
+                return value
+            if isinstance(value, str):
+                if value not in options:
+                    raise ValidationError(f"String value should be one of {options}")
+                return value
+
+            raise ValidationError("Field should be either an integer, tuple of integers, or a string")
+
+        def _jsonschema_type_mapping(self):
+            if None in options and not self.allow_none:
+                raise AssertionError(
+                    f"Provided string options `{options}` includes `None`, but field is not set to allow `None`."
+                )
+
+            # Prepare numeric option:
+            numeric_option = {
+                "type": "integer",
+                "title": "integer_option",
+                "default": default_integer,
+                "description": "Set to a valid number.",
+            }
+            tuple_option = {
+                "type": "array",
+                "title": "tuple_option",
+                "items": [{"type": "number", "minimum": 0, "maximum": 999999}] * 2,
+                "default": default_tuple,
+                "description": "Set to a valid number.",
+            }
+
+            # Prepare string option (remove None):
+            if None in options:
+                options.remove(None)
+            string_option = {
+                "type": "string",
+                "enum": options,
+                "default": default_option,
+                "title": "preconfigured_option",
+                "description": "Choose a preconfigured option.",
+            }
+            oneof_list = [
+                numeric_option,
+                tuple_option,
+                string_option,
+            ]
+
+            return {"oneOf": oneof_list, "title": self.name, "description": description, "default": default}
+
+    return field(
+        metadata={
+            "marshmallow_field": IntegerTupleStringOptionsField(
+                allow_none=allow_none, load_default=default, dump_default=default, metadata={"description": description}
+            )
+        },
+        default=default,
+    )
+
+
 def Dict(default: Union[None, TDict] = None, description: str = "", parameter_metadata: ParameterMetadata = None):
     """Returns a dataclass field with marshmallow metadata enforcing input must be a dict."""
     if default is not None:
@@ -436,6 +578,40 @@ def Dict(default: Union[None, TDict] = None, description: str = "", parameter_me
     )
 
 
+def List(
+    list_type: Union[Type[str], Type[int], Type[float]] = str, default: Union[None, TList[Any]] = None, description=""
+):
+    """Returns a dataclass field with marshmallow metadata enforcing input must be a list."""
+    if default is not None:
+        try:
+            assert isinstance(default, list)
+
+        except Exception:
+            raise ValidationError(f"Invalid default: `{default}`")
+
+    if list_type is str:
+        field_type = fields.String()
+    elif list_type is int:
+        field_type = fields.Integer()
+    elif list_type is float:
+        field_type = fields.Float()
+    else:
+        raise ValueError(f"Invalid list type: `{list_type}`")
+
+    return field(
+        metadata={
+            "marshmallow_field": fields.List(
+                field_type,
+                allow_none=True,
+                load_default=default,
+                dump_default=default,
+                metadata={"description": description},
+            )
+        },
+        default_factory=lambda: default,
+    )
+
+
 def DictList(
     default: Union[None, TList[TDict]] = None, description: str = "", parameter_metadata: ParameterMetadata = None
 ):
@@ -546,13 +722,6 @@ def _jsonschema_type_mapping(self):
             initializers = list(initializer_registry.keys())
             return {
                 "oneOf": [
-                    {
-                        "type": "string",
-                        "enum": initializers,
-                        "default": default,
-                        "title": "initializer_preconfigured_option",
-                        "description": "Pick a preconfigured initializer.",
-                    },
                     # Note: default not provided in the custom dict option:
                     {
                         "type": "object",
@@ -564,6 +733,13 @@ def _jsonschema_type_mapping(self):
                         "additionalProperties": True,
                         "description": "Customize an existing initializer.",
                     },
+                    {
+                        "type": "string",
+                        "enum": initializers,
+                        "default": default,
+                        "title": "initializer_preconfigured_option",
+                        "description": "Pick a preconfigured initializer.",
+                    },
                 ],
                 "title": self.name,
                 "default": default,
@@ -580,31 +756,42 @@ def _jsonschema_type_mapping(self):
     )
 
 
-def FloatRangeTupleDataclassField(N=2, default: Tuple = (0.9, 0.999), min=0, max=1, description=""):
-    """Returns a dataclass field with marshmallow metadata enforcing a `N`-dim. tuple with all values in given
-    range.
+def FloatRangeTupleDataclassField(
+    n=2, default: Tuple = (0.9, 0.999), allow_none: bool = False, min=0, max=1, description=""
+):
+    """Returns a dataclass field with marshmallow metadata enforcing a `N`-dim.
 
-    In particular, inputs must be N-dimensional tuples of purely numeric values within [min, max] range, i.e. inclusive.
-    The generated JSON schema uses a restricted array type as the equivalent representation of a Python tuple.
+    tuple with all values in given range. In particular, inputs must be N-dimensional tuples of purely numeric values
+    within [min, max] range, i.e. inclusive. The generated JSON schema uses a restricted array type as the equivalent
+    representation of a Python tuple.
     """
-    if N != len(default):
-        raise ValidationError(f"Dimension of tuple '{N}' must match dimension of default val. '{default}'")
+    if default is not None and n != len(default):
+        raise ValidationError(f"Dimension of tuple '{n}' must match dimension of default val. '{default}'")
 
     class FloatTupleMarshmallowField(fields.Tuple):
         def _jsonschema_type_mapping(self):
-            validate_range(default)
+            if default is not None:
+                validate_range(default)
             return {
-                "type": "array",
-                "items": [
+                "oneOf": [
                     {
-                        "type": "number",
-                        "minimum": min,
-                        "maximum": max,
-                    }
-                ]
-                * N,
+                        "type": "array",
+                        "items": [
+                            {
+                                "type": "number",
+                                "minimum": min,
+                                "maximum": max,
+                            }
+                        ]
+                        * n,
+                        "default": default,
+                        "description": description,
+                    },
+                    {"type": "null", "title": "null_float_tuple_option", "description": "None"},
+                ],
+                "title": self.name,
                 "default": default,
-                "description": description,
+                "description": "Valid options for FloatRangeTupleDataclassField.",
             }
 
     def validate_range(data: Tuple):
@@ -614,18 +801,21 @@ def validate_range(data: Tuple):
             raise ValidationError(
                 f"Values in received tuple should be in range [{min},{max}], instead received: {data}"
             )
-        raise ValidationError(f'Received value should be of {N}-dimensional "Tuple[float]", instead received: {data}')
+        raise ValidationError(f'Received value should be of {n}-dimensional "Tuple[float]", instead received: {data}')
 
     try:
-        validate_range(default)
+        if default is not None:
+            validate_range(default)
+        if default is None and not allow_none:
+            raise ValidationError("Default value must not be None if allow_none is False")
     except Exception:
         raise ValidationError(f"Invalid default: `{default}`")
 
     return field(
         metadata={
             "marshmallow_field": FloatTupleMarshmallowField(
-                tuple_fields=[fields.Float()] * N,
-                allow_none=False,
+                tuple_fields=[fields.Float()] * n,
+                allow_none=allow_none,
                 validate=validate_range,
                 load_default=default,
                 dump_default=default,
@@ -637,13 +827,10 @@ def validate_range(data: Tuple):
 
 
 def OneOfOptionsField(
-    default: TAny,
-    description: str,
-    allow_none: bool,
-    field_options: TList,
-    parameter_metadata: ParameterMetadata = None,
+    default: Any, description: str, allow_none: bool, field_options: TList, parameter_metadata: ParameterMetadata = None
 ):
     """Returns a dataclass field that is a combination of the other fields defined in `ludwig.schema.utils`."""
+    field_options_allow_none = any(option.metadata["marshmallow_field"].allow_none for option in field_options)
 
     class OneOfOptionsCombinatorialField(fields.Field):
         def _serialize(self, value, attr, obj, **kwargs):
@@ -687,16 +874,16 @@ def _jsonschema_type_mapping(self):
 
                     @m_dataclass
                     class DummyClass:
-                        tmp: TAny = option
+                        tmp: Any = option
 
                     dummy_schema = unload_jsonschema_from_marshmallow_class(DummyClass)
                     tmp_json_schema = dummy_schema["properties"]["tmp"]
                     oneOf["oneOf"].append(tmp_json_schema)
 
-            # Add null as an option if applicable:
+            # Add null as an option if none of the field options allow none:
             oneOf["oneOf"] += (
                 [{"type": "null", "title": "null_option", "description": "Disable this parameter."}]
-                if allow_none
+                if allow_none and not field_options_allow_none
                 else []
             )
 
diff --git a/ludwig/trainers/trainer_lightgbm.py b/ludwig/trainers/trainer_lightgbm.py
index 89f3bfc4fed..a1fb714e3f1 100644
--- a/ludwig/trainers/trainer_lightgbm.py
+++ b/ludwig/trainers/trainer_lightgbm.py
@@ -479,7 +479,7 @@ def _construct_lgb_params(self) -> Tuple[dict, dict]:
                 output_params = {
                     "objective": "multiclass",
                     "metric": ["multi_logloss"],
-                    "num_class": feature.num_classes,
+                    "num_class": feature.decoder_config.num_classes,
                 }
             elif feature.type() == BINARY:
                 output_params = {
diff --git a/ludwig/utils/backward_compatibility.py b/ludwig/utils/backward_compatibility.py
index 24db69d1207..37439f06aac 100644
--- a/ludwig/utils/backward_compatibility.py
+++ b/ludwig/utils/backward_compatibility.py
@@ -23,9 +23,11 @@
     COLUMN,
     CONV_BIAS,
     CONV_USE_BIAS,
+    DECODER,
     DEFAULT_BIAS,
     DEFAULT_USE_BIAS,
     DEFAULTS,
+    ENCODER,
     EVAL_BATCH_SIZE,
     EXECUTOR,
     FORCE_SPLIT,
@@ -53,6 +55,33 @@
 from ludwig.features.feature_registries import base_type_registry
 from ludwig.utils.misc_utils import merge_dict
 
+INPUT_FEATURE_KEYS = [
+    "name",
+    "type",
+    "column",
+    "proc_column",
+    "encoder",
+    "tied",
+    "preprocessing",
+    "vector_size",
+]
+
+OUTPUT_FEATURE_KEYS = [
+    "name",
+    "type",
+    "column",
+    "proc_column",
+    "decoder",
+    "tied",
+    "preprocessing",
+    "loss",
+    "reduce_input",
+    "dependencies",
+    "reduce_dependencies",
+    "top_k",
+    "vector_size",
+]
+
 
 def _traverse_dicts(config: Any, f: Callable[[Dict], None]):
     """Applies function f to every dictionary contained in config.
@@ -106,6 +135,49 @@ def _upgrade_feature(feature: Dict[str, Any]):
     _traverse_dicts(feature, _upgrade_use_bias)
 
 
+def _upgrade_encoder_decoder_params(feature: Dict[str, Any], input_feature: bool) -> None:
+    """
+    This function nests un-nested encoder/decoder parameters to conform with the new config structure for 0.6
+    Args:
+        feature (Dict): Feature to nest encoder/decoder params for.
+        input_feature (Bool): Whether this feature is an input feature or not.
+    """
+    warn = False
+    if input_feature:
+        module_type = ENCODER
+    else:
+        module_type = DECODER
+
+    module = feature.get(module_type, {})
+    keys = INPUT_FEATURE_KEYS if module_type == ENCODER else OUTPUT_FEATURE_KEYS + [ENCODER, DECODER]
+    if isinstance(module, str):
+        module = {TYPE: module}
+        feature[module_type] = module
+        warn = True
+
+    nested_params = []
+    for k, v in feature.items():
+        if k not in keys:
+            module[k] = v
+            nested_params.append(k)
+            warn = True
+
+    if module_type in feature:
+        feature[module_type].update(module)
+    else:
+        feature[module_type] = module
+
+    for k in nested_params:
+        del feature[k]
+
+    if warn:
+        warnings.warn(
+            f"{module_type} specific parameters should now be nested within a dictionary under the '{module_type}' "
+            f"parameter. Support for un-nested {module_type} specific parameters will be removed in v0.7",
+            DeprecationWarning,
+        )
+
+
 def _upgrade_hyperopt(hyperopt: Dict[str, Any]):
     """Upgrades hyperopt config (in-place)"""
     # check for use of legacy "training" reference, if any found convert to "trainer"
@@ -291,6 +363,12 @@ def upgrade_deprecated_fields(config: Dict[str, Any]):
     for feature in config.get(INPUT_FEATURES, []) + config.get(OUTPUT_FEATURES, []):
         _upgrade_feature(feature)
 
+    for feature in config.get(INPUT_FEATURES, []):
+        _upgrade_encoder_decoder_params(feature, input_feature=True)
+
+    for feature in config.get(OUTPUT_FEATURES, []):
+        _upgrade_encoder_decoder_params(feature, input_feature=False)
+
     if HYPEROPT in config:
         _upgrade_hyperopt(config[HYPEROPT])
 
diff --git a/ludwig/utils/checkpoint_utils.py b/ludwig/utils/checkpoint_utils.py
index 54bd9788d84..f7de76bf1aa 100644
--- a/ludwig/utils/checkpoint_utils.py
+++ b/ludwig/utils/checkpoint_utils.py
@@ -15,6 +15,7 @@
 
 from ludwig.utils.fs_utils import safe_move_file
 
+logger = logging.getLogger(__name__)
 LATEST_FNAME = "latest.ckpt"
 
 
@@ -84,7 +85,7 @@ def restore(self, save_path: str, device: Optional[torch.device] = None) -> bool
                 self.model.load_state_dict(state["model_weights"])
                 if self.optimizer is not None:
                     self.optimizer.load_state_dict(state["optim_state"])
-                logging.info(f"Successfully loaded model weights from {save_path}.")
+                logger.info(f"Successfully loaded model weights from {save_path}.")
                 return True
             except Exception as e:
                 # there was an issue loading the state which means
@@ -95,7 +96,7 @@ def restore(self, save_path: str, device: Optional[torch.device] = None) -> bool
                 # rather than allowing the program to proceed.
                 raise e
         except FileNotFoundError as e:
-            logging.error(e)
+            logger.error(e)
             return False
 
     def save(self, save_path: str, global_step: int):
@@ -134,7 +135,7 @@ def save(self, save_path: str, global_step: int):
                 torch.save(state, tmp_path)
 
                 safe_move_file(tmp_path, save_path)
-                logging.debug(f"Saved checkpoint at {save_path}.")
+                logger.debug(f"Saved checkpoint at {save_path}.")
         finally:
             # restore SIGINT handler
             if orig_handler is not None:
@@ -181,7 +182,7 @@ def restore_or_initialize(self) -> int:
         if last_ckpt:
             status = self.checkpoint.restore(last_ckpt, self.device)
             if not status:
-                logging.warning("Could not restore latest checkpoint file.")
+                logger.warning("Could not restore latest checkpoint file.")
                 return 0
             self.latest_checkpoint = last_ckpt
             return self.checkpoint.global_step
@@ -207,4 +208,4 @@ def load_latest_checkpoint(checkpoint: Checkpoint, directory: str, device: torch
         if last_ckpt:
             checkpoint.restore(last_ckpt, device)
         else:
-            logging.error(f"No checkpoints found in {directory}.")
+            logger.error(f"No checkpoints found in {directory}.")
diff --git a/ludwig/utils/config_utils.py b/ludwig/utils/config_utils.py
index 6ce162ee3a5..a0dbfc15863 100644
--- a/ludwig/utils/config_utils.py
+++ b/ludwig/utils/config_utils.py
@@ -1,6 +1,6 @@
 from typing import Any, Dict, Set, Union
 
-from ludwig.constants import INPUT_FEATURES, PREPROCESSING, TYPE
+from ludwig.constants import DECODER, ENCODER, INPUT_FEATURES, PREPROCESSING, TYPE
 from ludwig.features.feature_registries import input_type_registry, output_type_registry
 from ludwig.utils.misc_utils import get_from_registry
 
@@ -13,7 +13,12 @@ def get_feature_type_parameter_values_from_section(
     parameter_values = set()
     for feature in config[features_section]:
         if feature[TYPE] == feature_type:
-            parameter_values.add(feature[parameter_name])
+            if parameter_name in feature:
+                parameter_values.add(feature[parameter_name])
+            elif parameter_name in feature[ENCODER]:
+                parameter_values.add(feature[ENCODER][parameter_name])
+            elif parameter_name in feature[DECODER]:
+                parameter_values.add(feature[DECODER][parameter_name])
     return parameter_values
 
 
@@ -49,5 +54,7 @@ def merge_config_preprocessing_with_feature_specific_defaults(
 def get_default_encoder_or_decoder(feature: Dict[str, Any], config_feature_group: str) -> str:
     """Returns the default encoder or decoder for a feature."""
     if config_feature_group == INPUT_FEATURES:
-        return get_from_registry(feature.get(TYPE), input_type_registry).encoder
-    return get_from_registry(feature.get(TYPE), output_type_registry).decoder
+        feature_schema = get_from_registry(feature.get(TYPE), input_type_registry).get_schema_cls()
+        return feature_schema().encoder.type
+    feature_schema = get_from_registry(feature.get(TYPE), output_type_registry).get_schema_cls()
+    return feature_schema().decoder.type
diff --git a/ludwig/utils/defaults.py b/ludwig/utils/defaults.py
index 22125dd1384..28018a68dbb 100644
--- a/ludwig/utils/defaults.py
+++ b/ludwig/utils/defaults.py
@@ -56,7 +56,7 @@
 from ludwig.utils.config_utils import get_default_encoder_or_decoder, get_defaults_section_for_feature_type
 from ludwig.utils.data_utils import load_config_from_str, load_yaml
 from ludwig.utils.fs_utils import open_file
-from ludwig.utils.misc_utils import get_from_registry, merge_dict, set_default_value
+from ludwig.utils.misc_utils import get_from_registry, merge_dict, set_default_value, set_default_values
 from ludwig.utils.print_utils import print_ludwig
 
 logger = logging.getLogger(__name__)
@@ -225,7 +225,7 @@ def update_feature_from_defaults(config: Dict[str, Any], feature_dict: Dict[str,
         default_encoder_or_decoder = get_default_encoder_or_decoder(feature_dict, config_feature_group)
         if default_params_for_feature_type[TYPE] != default_encoder_or_decoder:
             # Update type and populate defaults for the encoder or decoder type
-            feature_dict[parameter] = default_params_for_feature_type[TYPE]
+            feature_dict[parameter] = default_params_for_feature_type
             get_from_registry(feature_dict[TYPE], registry_type).populate_defaults(feature_dict)
         # Make a copy of default encoder or decoder parameters without the type key.
         default_params_for_feature_type = copy.deepcopy(default_params_for_feature_type)
@@ -293,7 +293,7 @@ def merge_with_defaults(config: dict) -> dict:  # noqa: F821
     # ===== Input Features =====
     for input_feature in config[INPUT_FEATURES]:
         if config[MODEL_TYPE] == MODEL_GBM:
-            input_feature[ENCODER] = "passthrough"
+            set_default_values(input_feature, {ENCODER: {TYPE: "passthrough"}})
             remove_ecd_params(input_feature)
         get_from_registry(input_feature[TYPE], input_type_registry).populate_defaults(input_feature)
 
@@ -310,7 +310,7 @@ def merge_with_defaults(config: dict) -> dict:  # noqa: F821
     # ===== Output features =====
     for output_feature in config[OUTPUT_FEATURES]:
         if config[MODEL_TYPE] == MODEL_GBM:
-            output_feature[DECODER] = "passthrough"
+            set_default_values(output_feature, {DECODER: {TYPE: "passthrough"}})
             remove_ecd_params(output_feature)
         get_from_registry(output_feature[TYPE], output_type_registry).populate_defaults(output_feature)
 
diff --git a/ludwig/utils/misc_utils.py b/ludwig/utils/misc_utils.py
index 9f1f9cecff9..6c485131854 100644
--- a/ludwig/utils/misc_utils.py
+++ b/ludwig/utils/misc_utils.py
@@ -97,10 +97,24 @@ def set_default_value(dictionary, key, value):
         dictionary[key] = value
 
 
-def set_default_values(dictionary, default_value_dictionary):
-    # Set multiple default values
+def set_default_values(dictionary: dict, default_value_dictionary: dict):
+    """This function sets multiple default values recursively for various areas of the config. By using the helper
+    function set_default_value, It parses input values that contain nested dictionaries, only setting values for
+    parameters that have not already been defined by the user.
+
+    Args:
+        dictionary (dict): The dictionary to set default values for, generally a section of the config.
+        default_value_dictionary (dict): The dictionary containing the default values for the config.
+    """
     for key, value in default_value_dictionary.items():
-        set_default_value(dictionary, key, value)
+        if key not in dictionary:  # Event where the key is not in the dictionary yet
+            dictionary[key] = value
+        elif value == {}:  # Event where dict is empty
+            set_default_value(dictionary, key, value)
+        elif isinstance(value, dict) and value:  # Event where dictionary is nested - recursive call
+            set_default_values(dictionary[key], value)
+        else:
+            set_default_value(dictionary, key, value)
 
 
 def get_class_attributes(c):
diff --git a/ludwig/utils/strings_utils.py b/ludwig/utils/strings_utils.py
index bdbdc6dcb95..a3f1a8256c2 100644
--- a/ludwig/utils/strings_utils.py
+++ b/ludwig/utils/strings_utils.py
@@ -42,6 +42,8 @@
 PADDING_SYMBOL = "<PAD>"
 UNKNOWN_SYMBOL = "<UNK>"
 
+logger = logging.getLogger(__name__)
+
 
 class SpecialSymbol(Enum):
     """Special symbols used for text features."""
@@ -411,7 +413,7 @@ def build_sequence_matrix(
 
     max_length = processor.compute(unit_vectors.map(len).max())
     if max_length < length_limit:
-        logging.debug(f"max length of {format}: {max_length} < limit: {length_limit}")
+        logger.debug(f"max length of {format}: {max_length} < limit: {length_limit}")
     max_length = length_limit
 
     def pad(vector):
diff --git a/ludwig/visualize.py b/ludwig/visualize.py
index deb1eca5ae6..2af8358409a 100644
--- a/ludwig/visualize.py
+++ b/ludwig/visualize.py
@@ -108,7 +108,7 @@ def validate_conf_thresholds_and_probabilities_2d_3d(probabilities, threshold_ou
         item_len = len(value)
         if item_len != 2:
             exception_message = "Two {} should be provided - " "{} was given.".format(item, item_len)
-            logging.error(exception_message)
+            logger.error(exception_message)
             raise RuntimeError(exception_message)
 
 
@@ -130,7 +130,7 @@ def load_data_for_viz(load_type, model_file_statistics, **kwargs):
     try:
         stats_per_model = [loader(stats_f) for stats_f in model_file_statistics]
     except (TypeError, AttributeError):
-        logging.exception(f"Unable to open model statistics file {model_file_statistics}!")
+        logger.exception(f"Unable to open model statistics file {model_file_statistics}!")
         raise
     return stats_per_model
 
@@ -1847,7 +1847,7 @@ def compare_classifiers_multiclass_multimetric(
         for output_feature_name in output_feature_names:
             model_name_name = model_names_list[i] if model_names_list is not None and i < len(model_names_list) else ""
             if "per_class_stats" not in test_statistics[output_feature_name]:
-                logging.warning(
+                logger.warning(
                     f"The output_feature_name {output_feature_name} in test statistics does not contain "
                     + "per_class_stats, skipping it."
                 )
@@ -1932,22 +1932,22 @@ def compare_classifiers_multiclass_multimetric(
                     filename=filename,
                 )
 
-                logging.info("\n")
-                logging.info(model_name_name)
+                logger.info("\n")
+                logger.info(model_name_name)
                 tmp_str = f"{output_feature_name} best 5 classes: "
                 tmp_str += "{}"
-                logging.info(tmp_str.format(higher_f1s))
-                logging.info(f1_np[higher_f1s])
+                logger.info(tmp_str.format(higher_f1s))
+                logger.info(f1_np[higher_f1s])
                 tmp_str = f"{output_feature_name} worst 5 classes: "
                 tmp_str += "{}"
-                logging.info(tmp_str.format(lower_f1s))
-                logging.info(f1_np[lower_f1s])
+                logger.info(tmp_str.format(lower_f1s))
+                logger.info(f1_np[lower_f1s])
                 tmp_str = f"{output_feature_name} number of classes with f1 score > 0: "
                 tmp_str += "{}"
-                logging.info(tmp_str.format(np.sum(f1_np > 0)))
+                logger.info(tmp_str.format(np.sum(f1_np > 0)))
                 tmp_str = f"{output_feature_name} number of classes with f1 score = 0: "
                 tmp_str += "{}"
-                logging.info(tmp_str.format(np.sum(f1_np == 0)))
+                logger.info(tmp_str.format(np.sum(f1_np == 0)))
 
 
 def compare_classifiers_predictions(
diff --git a/tests/conftest.py b/tests/conftest.py
index 92f13137327..df10349d7ee 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -43,11 +43,11 @@ def yaml_filename():
 def hyperopt_results():
     """This function generates hyperopt results."""
     input_features = [
-        text_feature(name="utterance", cell_type="lstm", reduce_output="sum"),
-        category_feature(vocab_size=2, reduce_input="sum"),
+        text_feature(name="utterance", encoder={"cell_type": "lstm", "reduce_output": "sum"}),
+        category_feature(encoder={"vocab_size": 2}, reduce_input="sum"),
     ]
 
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     csv_filename = uuid.uuid4().hex[:10].upper() + ".csv"
     rel_path = generate_data(input_features, output_features, csv_filename)
diff --git a/tests/integration_tests/test_api.py b/tests/integration_tests/test_api.py
index f8a33f8d820..f049a2147d3 100644
--- a/tests/integration_tests/test_api.py
+++ b/tests/integration_tests/test_api.py
@@ -23,7 +23,7 @@
 
 from ludwig.api import LudwigModel
 from ludwig.callbacks import Callback
-from ludwig.constants import TRAINER
+from ludwig.constants import ENCODER, TRAINER, TYPE
 from ludwig.globals import MODEL_HYPERPARAMETERS_FILE_NAME
 from ludwig.models.inference import InferenceModule
 from ludwig.utils.data_utils import read_csv
@@ -151,31 +151,31 @@ def run_api_experiment_separated_datasets(input_features, output_features, data_
 
 def test_api_intent_classification(csv_filename):
     # Single sequence input, single category output
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
     for encoder in ENCODERS:
-        input_features[0]["encoder"] = encoder
+        input_features[0][ENCODER][TYPE] = encoder
         run_api_experiment(input_features, output_features, data_csv=rel_path)
 
 
 def test_api_intent_classification_separated(csv_filename):
     # Single sequence input, single category output
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
     for encoder in ENCODERS:
-        input_features[0]["encoder"] = encoder
+        input_features[0][ENCODER][TYPE] = encoder
         run_api_experiment_separated_datasets(input_features, output_features, data_csv=rel_path)
 
 
 def test_api_train_online(csv_filename):
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
     data_csv = generate_data(input_features, output_features, csv_filename)
 
     config = {
@@ -191,8 +191,8 @@ def test_api_train_online(csv_filename):
 
 
 def test_api_training_set(tmpdir):
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     data_csv = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
     val_csv = shutil.copyfile(data_csv, os.path.join(tmpdir, "validation.csv"))
@@ -212,8 +212,8 @@ def test_api_training_set(tmpdir):
 
 
 def test_api_training_determinism(tmpdir):
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     data_csv = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
 
@@ -351,8 +351,8 @@ def test_api_skip_parameters_train(
     skip_save_processed_input,
 ):
     # Single sequence input, single category output
-    input_features = [category_feature(vocab_size=5)]
-    output_features = [category_feature(vocab_size=5)]
+    input_features = [category_feature(encoder={"vocab_size": 5})]
+    output_features = [category_feature(decoder={"vocab_size": 5})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, csv_filename))
@@ -379,8 +379,8 @@ def test_api_skip_parameters_predict(
     skip_save_predictions,
 ):
     # Single sequence input, single category output
-    input_features = [category_feature(vocab_size=5)]
-    output_features = [category_feature(vocab_size=5)]
+    input_features = [category_feature(encoder={"vocab_size": 5})]
+    output_features = [category_feature(decoder={"vocab_size": 5})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, csv_filename))
@@ -409,8 +409,8 @@ def test_api_skip_parameters_evaluate(
     skip_collect_overall_stats,
 ):
     # Single sequence input, single category output
-    input_features = [category_feature(vocab_size=5)]
-    output_features = [category_feature(vocab_size=5)]
+    input_features = [category_feature(encoder={"vocab_size": 5})]
+    output_features = [category_feature(decoder={"vocab_size": 5})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, csv_filename))
@@ -438,8 +438,8 @@ def test_api_callbacks(tmpdir, csv_filename, epochs, batch_size, num_examples, s
     total_checkpoints = (steps_per_epoch / steps_per_checkpoint) * epochs
     total_batches = epochs * (num_examples / batch_size)
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     config = {
         "input_features": input_features,
@@ -487,8 +487,8 @@ def test_api_callbacks_checkpoints_per_epoch(
     total_checkpoints = epochs * checkpoints_per_epoch
     total_batches = epochs * (num_examples / batch_size)
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     config = {
         "input_features": input_features,
@@ -532,8 +532,8 @@ def test_api_callbacks_default_train_steps(tmpdir, csv_filename):
     num_examples = 80
     mock_callback = mock.Mock(wraps=Callback())
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     config = {
         "input_features": input_features,
@@ -559,8 +559,8 @@ def test_api_callbacks_fixed_train_steps(tmpdir, csv_filename):
     num_examples = 80
     mock_callback = mock.Mock(wraps=Callback())
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
     config = {
         "input_features": input_features,
         "output_features": output_features,
@@ -586,8 +586,8 @@ def test_api_callbacks_fixed_train_steps_less_than_one_epoch(tmpdir, csv_filenam
     num_examples = 80
     mock_callback = mock.Mock(wraps=Callback())
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
     config = {
         "input_features": input_features,
         "output_features": output_features,
@@ -615,8 +615,8 @@ def test_api_callbacks_fixed_train_steps_less_than_one_epoch(tmpdir, csv_filenam
 
 def test_api_save_torchscript(tmpdir):
     """Tests successful saving and loading of model in TorchScript format."""
-    input_features = [category_feature(vocab_size=5)]
-    output_features = [category_feature(name="class", vocab_size=5, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 5})]
+    output_features = [category_feature(name="class", encoder={"vocab_size": 5}, reduce_input="sum")]
 
     data_csv = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
     val_csv = shutil.copyfile(data_csv, os.path.join(tmpdir, "validation.csv"))
diff --git a/tests/integration_tests/test_audio_feature.py b/tests/integration_tests/test_audio_feature.py
index ad58d61a0c1..076035ce16d 100644
--- a/tests/integration_tests/test_audio_feature.py
+++ b/tests/integration_tests/test_audio_feature.py
@@ -10,16 +10,14 @@
 DEFAULT_OUTPUT_SIZE = 256
 
 
-@pytest.mark.parametrize(
-    "enc_encoder", ["stacked_cnn", "parallel_cnn", "stacked_parallel_cnn", "rnn", "cnnrnn", "passthrough"]
-)
+@pytest.mark.parametrize("enc_encoder", ["stacked_cnn", "parallel_cnn", "stacked_parallel_cnn", "rnn", "cnnrnn"])
 def test_audio_feature(enc_encoder):
     # synthetic audio tensor
     audio_tensor = torch.randn([BATCH_SIZE, SEQ_SIZE, AUDIO_W_SIZE], dtype=torch.float32)
 
     # generate audio feature config
     audio_feature_config = audio_feature(
-        folder=".", encoder=enc_encoder, max_sequence_length=SEQ_SIZE, embedding_size=AUDIO_W_SIZE
+        folder=".", encoder={"type": enc_encoder, "max_sequence_length": SEQ_SIZE, "embedding_size": AUDIO_W_SIZE}
     )
 
     # instantiate audio input feature object
diff --git a/tests/integration_tests/test_cache_manager.py b/tests/integration_tests/test_cache_manager.py
index a6da83edc02..93722b8adc8 100644
--- a/tests/integration_tests/test_cache_manager.py
+++ b/tests/integration_tests/test_cache_manager.py
@@ -42,8 +42,8 @@ def test_cache_dataset(use_cache_dir, use_split, use_df, tmpdir, change_test_dir
     assert manager.credentials == creds
 
     config = {
-        "input_features": [sequence_feature(reduce_output="sum")],
-        "output_features": [category_feature(vocab_size=2, reduce_input="sum")],
+        "input_features": [sequence_feature(encoder={"reduce_output": "sum"})],
+        "output_features": [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")],
         "combiner": {"type": "concat", "output_size": 14},
         "preprocessing": {},
     }
diff --git a/tests/integration_tests/test_cli.py b/tests/integration_tests/test_cli.py
index e41cd272b31..9b5e6e130a0 100644
--- a/tests/integration_tests/test_cli.py
+++ b/tests/integration_tests/test_cli.py
@@ -49,8 +49,8 @@ def _run_ludwig_horovod(command, **ludwig_kwargs):
 
 def _prepare_data(csv_filename, config_filename):
     # Single sequence input, single category output
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     dataset_filename = generate_data(input_features, output_features, csv_filename)
@@ -71,8 +71,8 @@ def _prepare_data(csv_filename, config_filename):
 
 def _prepare_hyperopt_data(csv_filename, config_filename):
     # Single sequence input, single category output
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     dataset_filename = generate_data(input_features, output_features, csv_filename)
diff --git a/tests/integration_tests/test_collect.py b/tests/integration_tests/test_collect.py
index 492ad771cf2..2d9dfdeb630 100644
--- a/tests/integration_tests/test_collect.py
+++ b/tests/integration_tests/test_collect.py
@@ -20,7 +20,7 @@
 
 from ludwig.api import LudwigModel
 from ludwig.collect import collect_activations, collect_weights, print_model_summary
-from ludwig.constants import TRAINER
+from ludwig.constants import ENCODER, TRAINER, TYPE
 from ludwig.utils.torch_utils import get_torch_device
 from tests.integration_tests.utils import category_feature, ENCODERS, generate_data, sequence_feature
 
@@ -29,10 +29,10 @@
 
 def _prepare_data(csv_filename):
     # Single sequence input, single category output
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
-    input_features[0]["encoder"] = ENCODERS[0]
+    input_features[0][ENCODER][TYPE] = ENCODERS[0]
 
     # Generate test data
     data_csv = generate_data(input_features, output_features, csv_filename)
diff --git a/tests/integration_tests/test_config_global_defaults.py b/tests/integration_tests/test_config_global_defaults.py
index 812b680fab0..67bb85aa49c 100644
--- a/tests/integration_tests/test_config_global_defaults.py
+++ b/tests/integration_tests/test_config_global_defaults.py
@@ -87,10 +87,10 @@ def test_global_default_parameters_merge_with_defaults(csv_filename):
 
     # Feature encoders and decoders should update
     for feature in updated_config[INPUT_FEATURES]:
-        assert feature[ENCODER] == updated_config[DEFAULTS][feature[TYPE]][ENCODER][TYPE]
+        assert feature[ENCODER][TYPE] == updated_config[DEFAULTS][feature[TYPE]][ENCODER][TYPE]
 
     output_feature = updated_config[OUTPUT_FEATURES][0]
-    assert output_feature[DECODER] == updated_config[DEFAULTS][output_feature[TYPE]][DECODER][TYPE]
+    assert output_feature[DECODER][TYPE] == updated_config[DEFAULTS][output_feature[TYPE]][DECODER][TYPE]
 
 
 def test_global_defaults_with_encoder_dependencies(csv_filename):
@@ -110,5 +110,5 @@ def test_global_defaults_with_encoder_dependencies(csv_filename):
     # Config should populate with the additional required fields for bert
     updated_config = merge_with_defaults(config)
 
-    assert updated_config[INPUT_FEATURES][0][ENCODER] == "bert"
+    assert updated_config[INPUT_FEATURES][0][ENCODER][TYPE] == "bert"
     assert updated_config[INPUT_FEATURES][0]["pretrained_model_name_or_path"] == "bert-base-uncased"
diff --git a/tests/integration_tests/test_custom_components.py b/tests/integration_tests/test_custom_components.py
index 8ca7ce61abc..184274a68c8 100644
--- a/tests/integration_tests/test_custom_components.py
+++ b/tests/integration_tests/test_custom_components.py
@@ -16,6 +16,8 @@
 from ludwig.modules.loss_modules import LogitsInputsMixin, register_loss
 from ludwig.modules.metric_modules import LossMetric, register_metric
 from ludwig.schema.combiners.base import BaseCombinerConfig
+from ludwig.schema.decoders.base import BaseDecoderConfig
+from ludwig.schema.encoders.base import BaseEncoderConfig
 from tests.integration_tests.utils import (
     category_feature,
     generate_data,
@@ -30,6 +32,22 @@ class CustomTestCombinerConfig(BaseCombinerConfig):
     foo: bool = False
 
 
+@dataclass
+class CustomNumberEncoderConfig(BaseEncoderConfig):
+
+    type: str = "custom_number_encoder"
+
+    input_size: int = 0
+
+
+@dataclass
+class CustomNumberDecoderConfig(BaseDecoderConfig):
+
+    type: str = "custom_number_decoder"
+
+    input_size: int = 0
+
+
 @register_combiner(name="custom_test")
 class CustomTestCombiner(Combiner):
     def __init__(self, input_features: Dict = None, config: CustomTestCombinerConfig = None, **kwargs):
@@ -69,6 +87,10 @@ def input_shape(self) -> torch.Size:
     def output_shape(self) -> torch.Size:
         return self.input_shape
 
+    @staticmethod
+    def get_schema_cls():
+        return CustomNumberEncoderConfig
+
 
 @register_decoder("custom_number_decoder", NUMBER)
 class CustomNumberDecoder(Decoder):
@@ -83,6 +105,10 @@ def input_shape(self):
     def forward(self, inputs, **kwargs):
         return torch.mean(inputs, 1)
 
+    @staticmethod
+    def get_schema_cls():
+        return CustomNumberDecoderConfig
+
 
 @register_loss("custom_loss", [NUMBER])
 class CustomLoss(nn.Module, LogitsInputsMixin):
@@ -109,11 +135,11 @@ def test_custom_combiner():
 
 def test_custom_encoder_decoder():
     input_features = [
-        sequence_feature(reduce_output="sum"),
-        number_feature(encoder="custom_number_encoder"),
+        sequence_feature(encoder={"reduce_output": "sum"}),
+        number_feature(encoder={"type": "custom_number_encoder"}),
     ]
     output_features = [
-        number_feature(decoder="custom_number_decoder"),
+        number_feature(decoder={"type": "custom_number_decoder"}),
     ]
     _run_test(input_features=input_features, output_features=output_features)
 
@@ -128,10 +154,10 @@ def test_custom_loss_metric():
 def _run_test(input_features=None, output_features=None, combiner=None):
     with tempfile.TemporaryDirectory() as tmpdir:
         input_features = input_features or [
-            sequence_feature(reduce_output="sum"),
+            sequence_feature(encoder={"reduce_output": "sum"}),
             number_feature(),
         ]
-        output_features = output_features or [category_feature(vocab_size=2, reduce_input="sum")]
+        output_features = output_features or [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
         combiner = combiner or {"type": "concat"}
 
         csv_filename = os.path.join(tmpdir, "training.csv")
diff --git a/tests/integration_tests/test_experiment.py b/tests/integration_tests/test_experiment.py
index 42ea320e078..0d826cbc979 100644
--- a/tests/integration_tests/test_experiment.py
+++ b/tests/integration_tests/test_experiment.py
@@ -25,7 +25,7 @@
 
 from ludwig.api import LudwigModel
 from ludwig.backend import LOCAL_BACKEND
-from ludwig.constants import H3, TRAINER
+from ludwig.constants import ENCODER, H3, TRAINER, TYPE
 from ludwig.data.concatenate_datasets import concatenate_df
 from ludwig.data.preprocessing import preprocess_for_training
 from ludwig.encoders.registry import get_encoder_classes
@@ -66,8 +66,10 @@
 
 @pytest.mark.parametrize("encoder", ENCODERS)
 def test_experiment_text_feature_non_HF(encoder, csv_filename):
-    input_features = [text_feature(vocab_size=30, min_len=1, encoder=encoder, preprocessing={"tokenizer": "space"})]
-    output_features = [category_feature(vocab_size=2)]
+    input_features = [
+        text_feature(encoder={"vocab_size": 30, "min_len": 1, "type": encoder}, preprocessing={"tokenizer": "space"})
+    ]
+    output_features = [category_feature(decoder={"vocab_size": 2})]
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
     run_experiment(input_features, output_features, dataset=rel_path)
@@ -76,14 +78,8 @@ def test_experiment_text_feature_non_HF(encoder, csv_filename):
 def run_experiment_with_encoder(encoder, csv_filename):
     # Run in a subprocess to clear TF and prevent OOM
     # This also allows us to use GPU resources
-    input_features = [
-        text_feature(
-            vocab_size=30,
-            min_len=1,
-            encoder=encoder,
-        )
-    ]
-    output_features = [category_feature(vocab_size=2)]
+    input_features = [text_feature(encoder={"vocab_size": 30, "min_len": 1, "type": encoder})]
+    output_features = [category_feature(decoder={"vocab_size": 2})]
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
     run_experiment(input_features, output_features, dataset=rel_path)
@@ -102,8 +98,8 @@ def test_experiment_text_feature_HF_full(encoder, csv_filename):
 
 @pytest.mark.parametrize("encoder", ENCODERS)
 def test_experiment_seq_seq_generator(csv_filename, encoder):
-    input_features = [text_feature(reduce_output=None, encoder=encoder)]
-    output_features = [text_feature(decoder="generator")]
+    input_features = [text_feature(encoder={"type": encoder, "reduce_output": None})]
+    output_features = [text_feature(decoder={"type": "generator"})]
     rel_path = generate_data(input_features, output_features, csv_filename)
 
     run_experiment(input_features, output_features, dataset=rel_path)
@@ -111,8 +107,8 @@ def test_experiment_seq_seq_generator(csv_filename, encoder):
 
 @pytest.mark.parametrize("encoder", ["embed", "rnn", "parallel_cnn", "stacked_parallel_cnn", "transformer"])
 def test_experiment_seq_seq_tagger(csv_filename, encoder):
-    input_features = [text_feature(reduce_output=None, encoder=encoder)]
-    output_features = [text_feature(decoder="tagger")]
+    input_features = [text_feature(encoder={"type": encoder, "reduce_output": None})]
+    output_features = [text_feature(decoder={"type": "tagger"})]
     rel_path = generate_data(input_features, output_features, csv_filename)
 
     run_experiment(input_features, output_features, dataset=rel_path)
@@ -120,8 +116,8 @@ def test_experiment_seq_seq_tagger(csv_filename, encoder):
 
 @pytest.mark.parametrize("encoder", ["cnnrnn", "stacked_cnn"])
 def test_experiment_seq_seq_tagger_fails_for_non_length_preserving_encoders(csv_filename, encoder):
-    input_features = [text_feature(reduce_output=None, encoder=encoder)]
-    output_features = [text_feature(decoder="tagger")]
+    input_features = [text_feature(encoder={"type": encoder, "reduce_output": None})]
+    output_features = [text_feature(decoder={"type": "tagger"})]
     rel_path = generate_data(input_features, output_features, csv_filename)
 
     with pytest.raises(ValueError):
@@ -130,8 +126,8 @@ def test_experiment_seq_seq_tagger_fails_for_non_length_preserving_encoders(csv_
 
 def test_experiment_seq_seq_model_def_file(csv_filename, yaml_filename):
     # seq-to-seq test to use config file instead of dictionary
-    input_features = [text_feature(reduce_output=None, encoder="embed")]
-    output_features = [text_feature(reduce_input=None, vocab_size=3, decoder="tagger")]
+    input_features = [text_feature(encoder={"reduce_output": None, "type": "embed"})]
+    output_features = [text_feature(decoder={"reduce_input": None, "vocab_size": 3, "type": "tagger"})]
 
     # Save the config to a yaml file
     config = {
@@ -149,8 +145,8 @@ def test_experiment_seq_seq_model_def_file(csv_filename, yaml_filename):
 
 def test_experiment_seq_seq_train_test_valid(tmpdir):
     # seq-to-seq test to use train, test, validation files
-    input_features = [text_feature(reduce_output=None, encoder="rnn")]
-    output_features = [text_feature(reduce_input=None, vocab_size=3, decoder="tagger")]
+    input_features = [text_feature(encoder={"reduce_output": None, "type": "rnn"})]
+    output_features = [text_feature(decoder={"reduce_input": None, "vocab_size": 3, "type": "tagger"})]
 
     train_csv = generate_data(input_features, output_features, os.path.join(tmpdir, "train.csv"))
     test_csv = generate_data(input_features, output_features, os.path.join(tmpdir, "test.csv"), 20)
@@ -169,13 +165,16 @@ def test_experiment_seq_seq_train_test_valid(tmpdir):
 @pytest.mark.parametrize("encoder", ENCODERS)
 def test_experiment_multi_input_intent_classification(csv_filename, encoder):
     # Multiple inputs, Single category output
-    input_features = [text_feature(vocab_size=10, min_len=1, representation="sparse"), category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [
+        text_feature(encoder={"vocab_size": 10, "min_len": 1, "representation": "sparse"}),
+        category_feature(encoder={"vocab_size": 10}),
+    ]
+    output_features = [category_feature(decoder={"reduce_input": "sum", "vocab_size": 2})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
 
-    input_features[0]["encoder"] = encoder
+    input_features[0][ENCODER][TYPE] = encoder
     run_experiment(input_features, output_features, dataset=rel_path)
 
 
@@ -189,8 +188,8 @@ def test_experiment_with_torch_module_dict_feature_name(csv_filename):
 
 def test_experiment_multiclass_with_class_weights(csv_filename):
     # Multiple inputs, Single category output
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=3, loss={"class_weights": [0, 1, 2, 3]})]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 3}, loss={"class_weights": [0, 1, 2, 3]})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -199,8 +198,8 @@ def test_experiment_multiclass_with_class_weights(csv_filename):
 
 def test_experiment_multilabel_with_class_weights(csv_filename):
     # Multiple inputs, Single category output
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [set_feature(vocab_size=3, loss={"class_weights": [0, 1, 2, 3]})]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [set_feature(decoder={"vocab_size": 3}, loss={"class_weights": [0, 1, 2, 3]})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -212,20 +211,20 @@ def test_experiment_multilabel_with_class_weights(csv_filename):
     [
         # baseline test case
         [
-            category_feature(vocab_size=2, reduce_input="sum"),
-            sequence_feature(vocab_size=10, max_len=5),
+            category_feature(decoder={"reduce_input": "sum", "vocab_size": 2}),
+            sequence_feature(decoder={"vocab_size": 10, "max_len": 5}),
             number_feature(),
         ],
         # use generator as decoder
         [
-            category_feature(vocab_size=2, reduce_input="sum"),
-            sequence_feature(vocab_size=10, max_len=5, decoder="generator"),
+            category_feature(decoder={"vocab_size": 2, "reduce_input": "sum"}),
+            sequence_feature(decoder={"vocab_size": 10, "max_len": 5, "type": "generator"}),
             number_feature(),
         ],
         # Generator decoder and reduce_input = None
         [
-            category_feature(vocab_size=2, reduce_input="sum"),
-            sequence_feature(max_len=5, decoder="generator", reduce_input=None),
+            category_feature(decoder={"vocab_size": 2, "reduce_input": "sum"}),
+            sequence_feature(decoder={"max_len": 5, "reduce_input": None, "type": "generator"}),
             number_feature(normalization="minmax"),
         ],
         # output features with dependencies single dependency
@@ -241,11 +240,11 @@ def test_experiment_multilabel_with_class_weights(csv_filename):
 )
 def test_experiment_multiple_seq_seq(csv_filename, output_features):
     input_features = [
-        text_feature(vocab_size=100, min_len=1, encoder="stacked_cnn"),
+        text_feature(encoder={"vocab_size": 100, "min_len": 1, "type": "stacked_cnn"}),
         number_feature(normalization="zscore"),
-        category_feature(vocab_size=10, embedding_size=5),
+        category_feature(encoder={"vocab_size": 10, "embedding_size": 5}),
         set_feature(),
-        sequence_feature(vocab_size=10, max_len=10, encoder="embed"),
+        sequence_feature(encoder={"vocab_size": 10, "max_len": 10, "type": "embed"}),
     ]
     output_features = output_features
 
@@ -264,7 +263,6 @@ def test_basic_image_feature(num_channels, image_source, in_memory, skip_save_pr
     input_features = [
         image_feature(
             folder=image_dest_folder,
-            encoder="stacked_cnn",
             preprocessing={
                 "in_memory": in_memory,
                 "height": 12,
@@ -272,11 +270,14 @@ def test_basic_image_feature(num_channels, image_source, in_memory, skip_save_pr
                 "num_channels": num_channels,
                 "num_processes": 5,
             },
-            output_size=16,
-            num_filters=8,
+            encoder={
+                "type": "stacked_cnn",
+                "output_size": 16,
+                "num_filters": 8,
+            },
         )
     ]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    output_features = [category_feature(decoder={"reduce_input": "sum", "vocab_size": 2})]
 
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
 
@@ -300,11 +301,11 @@ def test_experiment_infer_image_metadata(tmpdir):
 
     # Resnet encoder
     input_features = [
-        image_feature(folder=image_dest_folder, encoder="stacked_cnn", output_size=16, num_filters=8),
-        text_feature(encoder="embed", min_len=1),
+        image_feature(folder=image_dest_folder, encoder={"type": "stacked_cnn", "output_size": 16, "num_filters": 8}),
+        text_feature(encoder={"type": "embed", "min_len": 1}),
         number_feature(normalization="zscore"),
     ]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum"), number_feature()]
+    output_features = [category_feature(decoder={"reduce_input": "sum", "vocab_size": 2}), number_feature()]
 
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
 
@@ -333,17 +334,15 @@ def test_experiment_image_inputs(image_params: ImageParams, tmpdir):
     input_features = [
         image_feature(
             folder=image_dest_folder,
-            encoder="resnet",
             preprocessing={"in_memory": True, "height": 12, "width": 12, "num_channels": 3, "num_processes": 5},
-            output_size=16,
-            num_filters=8,
+            encoder={"type": "resnet", "output_size": 16, "num_filters": 8},
         ),
-        text_feature(encoder="embed", min_len=1),
+        text_feature(encoder={"type": "embed", "min_len": 1}),
         number_feature(normalization="zscore"),
     ]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum"), number_feature()]
+    output_features = [category_feature(decoder={"reduce_input": "sum", "vocab_size": 2}), number_feature()]
 
-    input_features[0]["encoder"] = image_params.image_encoder
+    input_features[0]["encoder"]["type"] = image_params.image_encoder
     input_features[0]["preprocessing"]["in_memory"] = image_params.in_memory_flag
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
 
@@ -368,14 +367,12 @@ def test_experiment_image_dataset(train_format, train_in_memory, test_format, te
     input_features = [
         image_feature(
             folder=image_dest_folder,
-            encoder="stacked_cnn",
             preprocessing={"in_memory": True, "height": 12, "width": 12, "num_channels": 3, "num_processes": 5},
-            output_size=16,
-            num_filters=8,
+            encoder={"type": "stacked_cnn", "output_size": 16, "num_filters": 8},
         ),
     ]
     output_features = [
-        category_feature(vocab_size=2, reduce_input="sum"),
+        category_feature(decoder={"reduce_input": "sum", "vocab_size": 2}),
     ]
 
     config = {
@@ -507,16 +504,18 @@ def test_experiment_audio_inputs(tmpdir):
 def test_experiment_tied_weights(csv_filename):
     # Single sequence input, single category output
     input_features = [
-        text_feature(name="text_feature1", min_len=1, encoder="cnnrnn", reduce_output="sum"),
-        text_feature(name="text_feature2", min_len=1, encoder="cnnrnn", reduce_output="sum", tied="text_feature1"),
+        text_feature(name="text_feature1", encoder={"min_len": 1, "type": "cnnrnn", "reduce_output": "sum"}),
+        text_feature(
+            name="text_feature2", encoder={"min_len": 1, "type": "cnnrnn", "reduce_output": "sum"}, tied="text_feature1"
+        ),
     ]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    output_features = [category_feature(decoder={"reduce_input": "sum", "vocab_size": 2})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
     for encoder in ENCODERS:
-        input_features[0]["encoder"] = encoder
-        input_features[1]["encoder"] = encoder
+        input_features[0][ENCODER][TYPE] = encoder
+        input_features[1][ENCODER][TYPE] = encoder
         run_experiment(input_features, output_features, dataset=rel_path)
 
 
@@ -524,8 +523,12 @@ def test_experiment_tied_weights(csv_filename):
 @pytest.mark.parametrize("attention", [False, True])
 def test_sequence_tagger(enc_cell_type, attention, csv_filename):
     # Define input and output features
-    input_features = [sequence_feature(max_len=10, encoder="rnn", cell_type=enc_cell_type, reduce_output=None)]
-    output_features = [sequence_feature(max_len=10, decoder="tagger", attention=attention, reduce_input=None)]
+    input_features = [
+        sequence_feature(encoder={"max_len": 10, "type": "rnn", "cell_type": enc_cell_type, "reduce_output": None})
+    ]
+    output_features = [
+        sequence_feature(decoder={"max_len": 10, "type": "tagger", "reduce_input": None, "attention": attention})
+    ]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -536,8 +539,8 @@ def test_sequence_tagger(enc_cell_type, attention, csv_filename):
 
 def test_sequence_tagger_text(csv_filename):
     # Define input and output features
-    input_features = [text_feature(max_len=10, encoder="rnn", reduce_output=None)]
-    output_features = [sequence_feature(max_len=10, decoder="tagger", reduce_input=None)]
+    input_features = [text_feature(encoder={"max_len": 10, "type": "rnn", "reduce_output": None})]
+    output_features = [sequence_feature(decoder={"max_len": 10, "reduce_input": None, "type": "tagger"})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -551,23 +554,27 @@ def test_experiment_sequence_combiner_with_reduction_fails(csv_filename):
         "input_features": [
             sequence_feature(
                 name="seq1",
-                min_len=5,
-                max_len=5,
-                encoder="embed",
-                cell_type="lstm",
-                reduce_output="sum",
+                encoder={
+                    "min_len": 5,
+                    "max_len": 5,
+                    "type": "embed",
+                    "cell_type": "lstm",
+                    "reduce_output": "sum",
+                },
             ),
             sequence_feature(
                 name="seq2",
-                min_len=5,
-                max_len=5,
-                encoder="embed",
-                cell_type="lstm",
-                reduce_output="sum",
+                encoder={
+                    "min_len": 5,
+                    "max_len": 5,
+                    "type": "embed",
+                    "cell_type": "lstm",
+                    "reduce_output": "sum",
+                },
             ),
-            category_feature(vocab_size=5),
+            category_feature(encoder={"vocab_size": 5}),
         ],
-        "output_features": [category_feature(reduce_input="sum", vocab_size=5)],
+        "output_features": [category_feature(decoder={"reduce_input": "sum", "vocab_size": 5})],
         TRAINER: {"epochs": 2},
         "combiner": {
             "type": "sequence",
@@ -591,14 +598,28 @@ def test_experiment_sequence_combiner(sequence_encoder, csv_filename):
     config = {
         "input_features": [
             sequence_feature(
-                name="seq1", min_len=5, max_len=5, encoder=sequence_encoder, cell_type="lstm", reduce_output=None
+                name="seq1",
+                encoder={
+                    "min_len": 5,
+                    "max_len": 5,
+                    "type": sequence_encoder,
+                    "cell_type": "lstm",
+                    "reduce_output": None,
+                },
             ),
             sequence_feature(
-                name="seq2", min_len=5, max_len=5, encoder=sequence_encoder, cell_type="lstm", reduce_output=None
+                name="seq2",
+                encoder={
+                    "min_len": 5,
+                    "max_len": 5,
+                    "type": sequence_encoder,
+                    "cell_type": "lstm",
+                    "reduce_output": None,
+                },
             ),
             category_feature(vocab_size=5),
         ],
-        "output_features": [category_feature(reduce_input="sum", vocab_size=5)],
+        "output_features": [category_feature(decoder={"reduce_input": "sum", "vocab_size": 5})],
         TRAINER: {"epochs": 2},
         "combiner": {
             "type": "sequence",
@@ -617,8 +638,8 @@ def test_experiment_sequence_combiner(sequence_encoder, csv_filename):
 def test_experiment_model_resume(tmpdir):
     # Single sequence input, single category output
     # Tests saving a model file, loading it to rerun training and predict
-    input_features = [sequence_feature(encoder="rnn", reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"type": "rnn", "reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"reduce_input": "sum", "vocab_size": 2})]
     # Generate test data
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
 
@@ -639,7 +660,7 @@ def test_experiment_model_resume(tmpdir):
 
 def test_experiment_various_feature_types(csv_filename):
     input_features = [binary_feature(), bag_feature()]
-    output_features = [set_feature(max_len=3, vocab_size=5)]
+    output_features = [set_feature(decoder={"max_len": 3, "vocab_size": 5})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -652,7 +673,7 @@ def test_experiment_timeseries(csv_filename):
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
-    input_features[0]["encoder"] = "transformer"
+    input_features[0][ENCODER][TYPE] = "transformer"
     run_experiment(input_features, output_features, dataset=rel_path)
 
 
@@ -661,14 +682,12 @@ def test_visual_question_answering(tmpdir):
     input_features = [
         image_feature(
             folder=image_dest_folder,
-            encoder="resnet",
             preprocessing={"in_memory": True, "height": 8, "width": 8, "num_channels": 3, "num_processes": 5},
-            output_size=8,
-            num_filters=8,
+            encoder={"type": "resnet", "output_size": 8, "num_filters": 8},
         ),
-        text_feature(encoder="embed", min_len=1),
+        text_feature(encoder={"type": "embed", "min_len": 1}),
     ]
-    output_features = [sequence_feature(decoder="generator", cell_type="lstm")]
+    output_features = [sequence_feature(decoder={"type": "generator", "cell_type": "lstm"})]
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
 
     run_experiment(input_features, output_features, dataset=rel_path)
@@ -688,12 +707,10 @@ def test_image_resizing_num_channel_handling(tmpdir):
     input_features = [
         image_feature(
             folder=image_dest_folder,
-            encoder="resnet",
             preprocessing={"in_memory": True, "height": 8, "width": 8, "num_channels": 3, "num_processes": 5},
-            output_size=8,
-            num_filters=8,
+            encoder={"type": "resnet", "output_size": 8, "num_filters": 8},
         ),
-        text_feature(encoder="embed", min_len=1),
+        text_feature(encoder={"type": "embed", "min_len": 1}),
         number_feature(normalization="minmax"),
     ]
     output_features = [binary_feature(), number_feature()]
@@ -720,12 +737,12 @@ def test_image_resizing_num_channel_handling(tmpdir):
 @pytest.mark.parametrize("encoder", ["wave", "embed"])
 def test_experiment_date(encoder, csv_filename):
     input_features = [date_feature()]
-    output_features = [category_feature(vocab_size=2)]
+    output_features = [category_feature(decoder={"vocab_size": 2})]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
 
-    input_features[0]["encoder"] = encoder
+    input_features[0][ENCODER] = {TYPE: encoder}
     run_experiment(input_features, output_features, dataset=rel_path)
 
 
@@ -737,7 +754,7 @@ def test_experiment_h3(encoder, csv_filename):
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
 
-    input_features[0]["encoder"] = encoder
+    input_features[0][ENCODER] = {TYPE: encoder}
     run_experiment(input_features, output_features, dataset=rel_path)
 
 
diff --git a/tests/integration_tests/test_gbm.py b/tests/integration_tests/test_gbm.py
index 63345b25114..1ce6cdf097c 100644
--- a/tests/integration_tests/test_gbm.py
+++ b/tests/integration_tests/test_gbm.py
@@ -55,7 +55,7 @@ def ray_backend():
 
 def run_test_gbm_output_not_supported(tmpdir, backend_config):
     """Test that an error is raised when the output feature is not supported by the model."""
-    input_features = [number_feature(), category_feature(reduce_output="sum")]
+    input_features = [number_feature(), category_feature(encoder={"reduce_output": "sum"})]
     output_features = [text_feature()]
 
     csv_filename = os.path.join(tmpdir, "training.csv")
@@ -82,11 +82,11 @@ def test_ray_gbm_output_not_supported(tmpdir, ray_backend):
 
 def run_test_gbm_multiple_outputs(tmpdir, backend_config):
     """Test that an error is raised when the model is trained with multiple outputs."""
-    input_features = [number_feature(), category_feature(reduce_output="sum")]
+    input_features = [number_feature(), category_feature(encoder={"reduce_output": "sum"})]
     output_features = [
-        category_feature(vocab_size=3),
+        category_feature(decoder={"vocab_size": 3}),
         binary_feature(),
-        category_feature(vocab_size=3),
+        category_feature(decoder={"vocab_size": 3}),
     ]
 
     csv_filename = os.path.join(tmpdir, "training.csv")
@@ -116,7 +116,7 @@ def test_ray_gbm_multiple_outputs(tmpdir, ray_backend):
 
 def run_test_gbm_binary(tmpdir, backend_config):
     """Test that the GBM model can train and predict a binary variable (binary classification)."""
-    input_features = [number_feature(), category_feature(reduce_output="sum")]
+    input_features = [number_feature(), category_feature(encoder={"reduce_output": "sum"})]
     output_feature = binary_feature()
     output_features = [output_feature]
 
@@ -162,9 +162,9 @@ def test_ray_gbm_binary(tmpdir, ray_backend):
 
 def run_test_gbm_category(tmpdir, backend_config):
     """Test that the GBM model can train and predict a categorical output (multiclass classification)."""
-    input_features = [number_feature(), category_feature(reduce_output="sum")]
+    input_features = [number_feature(), category_feature(encoder={"reduce_output": "sum"})]
     vocab_size = 3
-    output_feature = category_feature(vocab_size=vocab_size)
+    output_feature = category_feature(decoder={"vocab_size": vocab_size})
     output_features = [output_feature]
 
     csv_filename = os.path.join(tmpdir, "training.csv")
@@ -211,7 +211,7 @@ def test_ray_gbm_category(tmpdir, ray_backend):
 def run_test_gbm_number(tmpdir, backend_config):
     """Test that the GBM model can train and predict a numerical output (regression)."""
     # Given a dataset with a single input feature and a single output feature,
-    input_features = [number_feature(), category_feature(reduce_output="sum")]
+    input_features = [number_feature(), category_feature(encoder={"reduce_output": "sum"})]
     output_feature = number_feature()
     output_features = [output_feature]
 
diff --git a/tests/integration_tests/test_graph_execution.py b/tests/integration_tests/test_graph_execution.py
index 3de63b2ec29..a7f3ffb2a21 100644
--- a/tests/integration_tests/test_graph_execution.py
+++ b/tests/integration_tests/test_graph_execution.py
@@ -31,20 +31,20 @@
     [
         # baseline test case
         [
-            category_feature(vocab_size=2, reduce_input="sum"),
-            sequence_feature(vocab_size=10, max_len=5),
+            category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
+            sequence_feature(decoder={"vocab_size": 10, "max_len": 5}),
             number_feature(),
         ],
         # use generator as decoder
         [
-            category_feature(vocab_size=2, reduce_input="sum"),
-            sequence_feature(vocab_size=10, max_len=5, decoder="generator"),
+            category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
+            sequence_feature(decoder={"vocab_size": 10, "max_len": 5, "type": "generator"}),
             number_feature(),
         ],
         # Generator decoder and reduce_input = None
         [
-            category_feature(vocab_size=2, reduce_input="sum"),
-            sequence_feature(max_len=5, decoder="generator", reduce_input=None),
+            category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
+            sequence_feature(decoder={"max_len": 5, "type": "generator"}, reduce_input=None),
             number_feature(normalization="minmax"),
         ],
         # output features with dependencies single dependency
@@ -55,11 +55,11 @@
 )
 def test_experiment_multiple_seq_seq(csv_filename, output_features):
     input_features = [
-        text_feature(vocab_size=100, min_len=1, encoder="stacked_cnn"),
+        text_feature(encoder={"vocab_size": 100, "min_len": 1, "type": "stacked_cnn"}),
         number_feature(normalization="zscore"),
-        category_feature(vocab_size=10, embedding_size=5),
+        category_feature(encoder={"vocab_size": 10, "embedding_size": 5}),
         set_feature(),
-        sequence_feature(vocab_size=10, max_len=10, encoder="embed"),
+        sequence_feature(encoder={"vocab_size": 10, "max_len": 10, "type": "embed"}),
     ]
     output_features = output_features
 
diff --git a/tests/integration_tests/test_horovod.py b/tests/integration_tests/test_horovod.py
index f659e5677d8..d4b8d3e6313 100644
--- a/tests/integration_tests/test_horovod.py
+++ b/tests/integration_tests/test_horovod.py
@@ -29,6 +29,7 @@
 else:
     HOROVOD_AVAILABLE = True
 
+from ludwig.constants import ENCODER, TYPE
 from tests.integration_tests.utils import category_feature, ENCODERS, generate_data, sequence_feature
 
 # This script will run the actual test model training in parallel
@@ -71,10 +72,10 @@ def _run_horovod(csv_filename, **ludwig_kwargs):
 
 def _prepare_data(csv_filename):
     # Single sequence input, single category output
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
-    input_features[0]["encoder"] = ENCODERS[0]
+    input_features[0][ENCODER][TYPE] = ENCODERS[0]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
diff --git a/tests/integration_tests/test_hyperopt.py b/tests/integration_tests/test_hyperopt.py
index ff01e324b87..f89995dec5e 100644
--- a/tests/integration_tests/test_hyperopt.py
+++ b/tests/integration_tests/test_hyperopt.py
@@ -15,7 +15,7 @@
 import contextlib
 import json
 import os.path
-from typing import Dict, Optional, Tuple, Union
+from typing import Any, Dict, Optional, Tuple, Union
 
 import pytest
 import torch
@@ -25,6 +25,8 @@
     ACCURACY,
     CATEGORY,
     COMBINER,
+    DECODER,
+    ENCODER,
     HYPEROPT,
     INPUT_FEATURES,
     NAME,
@@ -101,11 +103,11 @@
 
 def _setup_ludwig_config(dataset_fp: str) -> Tuple[Dict, str]:
     input_features = [
-        text_feature(name="utterance", reduce_output="sum"),
-        category_feature(vocab_size=3),
+        text_feature(name="utterance", encoder={"reduce_output": "sum"}),
+        category_feature(encoder={"vocab_size": 3}),
     ]
 
-    output_features = [category_feature(vocab_size=3)]
+    output_features = [category_feature(decoder={"vocab_size": 3})]
 
     rel_path = generate_data(input_features, output_features, dataset_fp)
 
@@ -121,20 +123,21 @@ def _setup_ludwig_config(dataset_fp: str) -> Tuple[Dict, str]:
     return config, rel_path
 
 
-def _setup_ludwig_config_with_shared_params(dataset_fp: str) -> Tuple[Dict, str]:
+def _setup_ludwig_config_with_shared_params(dataset_fp: str) -> Tuple[Dict, Any]:
     input_features = [
-        text_feature(name="title", encoder="parallel_cnn"),
+        text_feature(name="title", encoder={"type": "parallel_cnn"}),
         text_feature(name="summary"),
-        category_feature(vocab_size=3),
-        category_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
+        category_feature(encoder={"vocab_size": 3}),
     ]
 
-    output_features = [category_feature(vocab_size=3)]
+    output_features = [category_feature(decoder={"vocab_size": 3})]
 
     rel_path = generate_data(input_features, output_features, dataset_fp)
 
     num_filters_search_space = [4, 8]
     embedding_size_search_space = [4, 8]
+    reduce_input_search_space = ["sum", "mean"]
 
     # Add default parameters in hyperopt parameter search space
     config = {
@@ -145,11 +148,15 @@ def _setup_ludwig_config_with_shared_params(dataset_fp: str) -> Tuple[Dict, str]
         HYPEROPT: {
             "parameters": {
                 "trainer.learning_rate": {"lower": 0.0001, "upper": 0.01, "space": "loguniform"},
-                "defaults.text.num_filters": {"space": "choice", "categories": num_filters_search_space},
-                "defaults.category.embedding_size": {
+                "defaults.text.encoder.num_filters": {"space": "choice", "categories": num_filters_search_space},
+                "defaults.category.encoder.embedding_size": {
                     "space": "choice",
                     "categories": embedding_size_search_space,
                 },
+                "defaults.category.decoder.reduce_input": {
+                    "space": "choice",
+                    "categories": reduce_input_search_space,
+                },
             },
             "goal": "minimize",
             "output_feature": output_features[0][NAME],
@@ -159,7 +166,7 @@ def _setup_ludwig_config_with_shared_params(dataset_fp: str) -> Tuple[Dict, str]
         },
     }
 
-    return config, rel_path, num_filters_search_space, embedding_size_search_space
+    return config, rel_path, num_filters_search_space, embedding_size_search_space, reduce_input_search_space
 
 
 def _get_trial_parameter_value(parameter_key: str, trial_row: str) -> Union[str, None]:
@@ -311,11 +318,11 @@ def test_hyperopt_scheduler(
 @pytest.mark.parametrize("search_space", ["random", "grid"])
 def test_hyperopt_run_hyperopt(csv_filename, search_space, tmpdir, ray_cluster):
     input_features = [
-        text_feature(name="utterance", reduce_output="sum"),
-        category_feature(vocab_size=3),
+        text_feature(name="utterance", encoder={"reduce_output": "sum"}),
+        category_feature(encoder={"vocab_size": 3}),
     ]
 
-    output_features = [category_feature(vocab_size=3)]
+    output_features = [category_feature(decoder={"vocab_size": 3})]
 
     rel_path = generate_data(input_features, output_features, csv_filename)
 
@@ -386,18 +393,20 @@ def test_hyperopt_run_hyperopt(csv_filename, search_space, tmpdir, ray_cluster):
 
 
 def _test_hyperopt_with_shared_params_trial_table(
-    hyperopt_results_df, num_filters_search_space, embedding_size_search_space
+    hyperopt_results_df, num_filters_search_space, embedding_size_search_space, reduce_input_search_space
 ):
     # Check that hyperopt trials sample from defaults in the search space
     for _, trial_row in hyperopt_results_df.iterrows():
-        embedding_size = _get_trial_parameter_value("defaults.category.embedding_size", trial_row)
-        num_filters = _get_trial_parameter_value("defaults.text.num_filters", trial_row)
+        embedding_size = _get_trial_parameter_value("defaults.category.encoder.embedding_size", trial_row)
+        num_filters = _get_trial_parameter_value("defaults.text.encoder.num_filters", trial_row)
+        reduce_input = _get_trial_parameter_value("defaults.category.decoder.reduce_input", trial_row).replace('"', "")
         assert embedding_size in embedding_size_search_space
         assert num_filters in num_filters_search_space
+        assert reduce_input in reduce_input_search_space
 
 
 def _test_hyperopt_with_shared_params_written_config(
-    hyperopt_results_df, num_filters_search_space, embedding_size_search_space
+    hyperopt_results_df, num_filters_search_space, embedding_size_search_space, reduce_input_search_space
 ):
     # Check that each hyperopt trial's written input/output configs got updated
     for _, trial_row in hyperopt_results_df.iterrows():
@@ -408,9 +417,9 @@ def _test_hyperopt_with_shared_params_written_config(
         # Check that num_filters got updated from the sampler correctly
         for input_feature in model_parameters[INPUT_FEATURES]:
             if input_feature[TYPE] == TEXT:
-                assert input_feature["num_filters"] in num_filters_search_space
+                assert input_feature[ENCODER]["num_filters"] in num_filters_search_space
             elif input_feature[TYPE] == CATEGORY:
-                assert input_feature["embedding_size"] in embedding_size_search_space
+                assert input_feature[ENCODER]["embedding_size"] in embedding_size_search_space
 
         # All text features with defaults should have the same num_filters for this trial
         text_input_num_filters = get_feature_type_parameter_values_from_section(
@@ -418,35 +427,35 @@ def _test_hyperopt_with_shared_params_written_config(
         )
         assert len(text_input_num_filters) == 1
 
-        # Check that embedding_size got updated from the sampler
         for output_feature in model_parameters[OUTPUT_FEATURES]:
             if output_feature[TYPE] == CATEGORY:
-                assert output_feature["embedding_size"] in embedding_size_search_space
+                assert output_feature[DECODER]["reduce_input"] in reduce_input_search_space
 
         # All category features with defaults should have the same embedding_size for this trial
         input_category_features_embedding_sizes = get_feature_type_parameter_values_from_section(
             model_parameters, INPUT_FEATURES, CATEGORY, "embedding_size"
         )
-        output_category_features_embedding_sizes = get_feature_type_parameter_values_from_section(
-            model_parameters, OUTPUT_FEATURES, CATEGORY, "embedding_size"
-        )
-        trial_embedding_sizes = input_category_features_embedding_sizes.union(output_category_features_embedding_sizes)
-        assert len(trial_embedding_sizes) == 1
+
+        assert len(input_category_features_embedding_sizes) == 1
 
 
 @pytest.mark.distributed
 def test_hyperopt_with_shared_params(csv_filename, tmpdir):
-    config, rel_path, num_filters_search_space, embedding_size_search_space = _setup_ludwig_config_with_shared_params(
-        csv_filename
-    )
+    (
+        config,
+        rel_path,
+        num_filters_search_space,
+        embedding_size_search_space,
+        reduce_input_search_space,
+    ) = _setup_ludwig_config_with_shared_params(csv_filename)
 
     hyperopt_results = hyperopt(config, dataset=rel_path, output_directory=tmpdir, experiment_name="test_hyperopt")
     hyperopt_results_df = hyperopt_results.experiment_analysis.results_df
 
     _test_hyperopt_with_shared_params_trial_table(
-        hyperopt_results_df, num_filters_search_space, embedding_size_search_space
+        hyperopt_results_df, num_filters_search_space, embedding_size_search_space, reduce_input_search_space
     )
 
     _test_hyperopt_with_shared_params_written_config(
-        hyperopt_results_df, num_filters_search_space, embedding_size_search_space
+        hyperopt_results_df, num_filters_search_space, embedding_size_search_space, reduce_input_search_space
     )
diff --git a/tests/integration_tests/test_hyperopt_ray.py b/tests/integration_tests/test_hyperopt_ray.py
index aff08314bca..44aa559f4ef 100644
--- a/tests/integration_tests/test_hyperopt_ray.py
+++ b/tests/integration_tests/test_hyperopt_ray.py
@@ -84,11 +84,11 @@
 
 def _get_config(search_alg, executor):
     input_features = [
-        text_feature(name="utterance", cell_type="lstm", reduce_output="sum"),
-        category_feature(vocab_size=2, reduce_input="sum"),
+        text_feature(name="utterance", encoder={"cell_type": "lstm", "reduce_output": "sum"}),
+        category_feature(encoder={"vocab_size": 2}, reduce_input="sum"),
     ]
 
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     return {
         "input_features": input_features,
@@ -199,11 +199,11 @@ def test_hyperopt_executor_with_metric(use_split, csv_filename, tmpdir, ray_clus
 @pytest.mark.distributed
 def test_hyperopt_run_hyperopt(csv_filename, tmpdir, ray_cluster_4cpu):
     input_features = [
-        text_feature(name="utterance", cell_type="lstm", reduce_output="sum"),
-        category_feature(vocab_size=2, reduce_input="sum"),
+        text_feature(name="utterance", encoder={"cell_type": "lstm", "reduce_output": "sum"}),
+        category_feature(encoder={"vocab_size": 2}, reduce_input="sum"),
     ]
 
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     rel_path = generate_data(input_features, output_features, csv_filename)
 
diff --git a/tests/integration_tests/test_input_feature_tied.py b/tests/integration_tests/test_input_feature_tied.py
index d9d18162799..bc598e5385a 100644
--- a/tests/integration_tests/test_input_feature_tied.py
+++ b/tests/integration_tests/test_input_feature_tied.py
@@ -2,7 +2,7 @@
 
 import pytest
 
-from ludwig.models.ecd import ECD
+from ludwig.models.base import BaseModel
 from tests.integration_tests.utils import (
     category_feature,
     generate_data,
@@ -27,44 +27,82 @@
     "input_feature_options",
     [
         # tie input features, encoders should be the same
-        InputFeatureOptions("number", None, True),
-        InputFeatureOptions("number", {"preprocessing": {"normalization": "zscore"}}, True),
-        InputFeatureOptions("binary", None, True),
-        InputFeatureOptions("category", {"vocab": ["a", "b", "c"]}, True),
-        InputFeatureOptions("set", {"vocab": ["a", "b", "c"]}, True),
-        InputFeatureOptions("sequence", {"max_sequence_length": 10, "vocab": ["x", "y", "z"]}, True),
-        InputFeatureOptions("text", {"max_sequence_length": 10, "vocab": ["a", "b", "c"]}, True),
-        InputFeatureOptions("timeseries", {"max_sequence_length": 10, "should_embed": False}, True),
-        InputFeatureOptions("audio", {"embedding_size": 64, "max_sequence_length": 16, "should_embed": False}, True),
+        InputFeatureOptions("number", {"encoder": {"type": "passthrough"}}, True),
+        InputFeatureOptions(
+            "number", {"encoder": {"type": "passthrough"}, "preprocessing": {"normalization": "zscore"}}, True
+        ),
+        InputFeatureOptions("binary", {"encoder": {"type": "passthrough"}}, True),
+        InputFeatureOptions("category", {"encoder": {"type": "dense", "vocab": ["a", "b", "c"]}}, True),
+        InputFeatureOptions("set", {"encoder": {"type": "embed", "vocab": ["a", "b", "c"]}}, True),
+        InputFeatureOptions(
+            "sequence", {"encoder": {"type": "parallel_cnn", "max_sequence_length": 10, "vocab": ["x", "y", "z"]}}, True
+        ),
+        InputFeatureOptions(
+            "text", {"encoder": {"type": "parallel_cnn", "max_sequence_length": 10, "vocab": ["a", "b", "c"]}}, True
+        ),
+        InputFeatureOptions(
+            "timeseries", {"encoder": {"type": "parallel_cnn", "max_sequence_length": 10, "should_embed": False}}, True
+        ),
+        InputFeatureOptions(
+            "audio",
+            {
+                "encoder": {
+                    "type": "parallel_cnn",
+                    "embedding_size": 64,
+                    "max_sequence_length": 16,
+                    "should_embed": False,
+                }
+            },
+            True,
+        ),
         # do not tie input features, encoders should be different
-        InputFeatureOptions("number", None, False),
-        InputFeatureOptions("number", {"preprocessing": {"normalization": "zscore"}}, False),
-        InputFeatureOptions("binary", None, False),
-        InputFeatureOptions("category", {"vocab": ["a", "b", "c"]}, False),
-        InputFeatureOptions("set", {"vocab": ["a", "b", "c"]}, False),
-        InputFeatureOptions("sequence", {"max_sequence_length": 10, "vocab": ["x", "y", "z"]}, False),
-        InputFeatureOptions("text", {"max_sequence_length": 10, "vocab": ["a", "b", "c"]}, False),
-        InputFeatureOptions("timeseries", {"max_sequence_length": 10, "should_embed": False}, False),
-        InputFeatureOptions("audio", {"embedding_size": 64, "max_sequence_length": 16, "should_embed": False}, False),
+        InputFeatureOptions("number", {"encoder": {"type": "passthrough"}}, False),
+        InputFeatureOptions(
+            "number", {"encoder": {"type": "passthrough"}, "preprocessing": {"normalization": "zscore"}}, False
+        ),
+        InputFeatureOptions("binary", {"encoder": {"type": "passthrough"}}, False),
+        InputFeatureOptions("category", {"encoder": {"type": "dense", "vocab": ["a", "b", "c"]}}, False),
+        InputFeatureOptions("set", {"encoder": {"type": "embed", "vocab": ["a", "b", "c"]}}, False),
+        InputFeatureOptions(
+            "sequence",
+            {"encoder": {"type": "parallel_cnn", "max_sequence_length": 10, "vocab": ["x", "y", "z"]}},
+            False,
+        ),
+        InputFeatureOptions(
+            "text", {"encoder": {"type": "parallel_cnn", "max_sequence_length": 10, "vocab": ["a", "b", "c"]}}, False
+        ),
+        InputFeatureOptions(
+            "timeseries", {"encoder": {"type": "parallel_cnn", "max_sequence_length": 10, "should_embed": False}}, False
+        ),
+        InputFeatureOptions(
+            "audio",
+            {
+                "encoder": {
+                    "type": "parallel_cnn",
+                    "embedding_size": 64,
+                    "max_sequence_length": 16,
+                    "should_embed": False,
+                }
+            },
+            False,
+        ),
     ],
 )
 def test_tied_micro_level(input_feature_options):
     # build input feature config
-    input_feature_configs = []
+    input_feature_configs = list()
 
     input_feature_configs.append({"name": "input_feature_1", "type": input_feature_options.feature_type})
-    if input_feature_options.feature_options is not None:
-        input_feature_configs[0].update(input_feature_options.feature_options)
+    input_feature_configs[0].update(input_feature_options.feature_options)
 
     input_feature_configs.append({"name": "input_feature_2", "type": input_feature_options.feature_type})
-    if input_feature_options.feature_options is not None:
-        input_feature_configs[1].update(input_feature_options.feature_options)
+    input_feature_configs[1].update(input_feature_options.feature_options)
 
     # add tied option to the second feature
     if input_feature_options.tie_features:
         input_feature_configs[1]["tied"] = "input_feature_1"
 
-    input_features = ECD.build_inputs(input_feature_configs)
+    input_features = BaseModel.build_inputs(input_feature_configs)
 
     if input_feature_options.tie_features:
         # should be same encoder
diff --git a/tests/integration_tests/test_kfold_cv.py b/tests/integration_tests/test_kfold_cv.py
index e0b22fd9e94..2ff04e101d5 100644
--- a/tests/integration_tests/test_kfold_cv.py
+++ b/tests/integration_tests/test_kfold_cv.py
@@ -43,7 +43,7 @@
         # input feature
         [number_feature(normalization="zscore"), number_feature(normalization="zscore")],
         # output feature
-        [category_feature(vocab_size=4, reduce_input="sum")],
+        [category_feature(encoder={"vocab_size": 4}, reduce_input="sum")],
     ),
     FeaturesToUse(
         # input feature
@@ -52,15 +52,26 @@
         # output feature
         [
             sequence_feature(
-                min_len=5, max_len=10, decoder="generator", cell_type="lstm", attention="bahdanau", reduce_input=None
+                decoder={
+                    "min_len": 5,
+                    "max_len": 10,
+                    "type": "generator",
+                    "cell_type": "lstm",
+                    "attention": "bahdanau",
+                },
+                reduce_input=None,
             )
         ],
     ),
     FeaturesToUse(
         # input feature
-        [sequence_feature(min_len=5, max_len=10, encoder="rnn", cell_type="lstm", reduce_output=None)],
+        [
+            sequence_feature(
+                encoder={"min_len": 5, "max_len": 10, "encoder": "rnn", "cell_type": "lstm", "reduce_output": None}
+            )
+        ],
         # output feature
-        [sequence_feature(max_len=10, decoder="tagger", reduce_input=None)],
+        [sequence_feature(decoder={"max_len": 10, "decoder": "tagger"}, reduce_input=None)],
     ),
     FeaturesToUse(
         # input feature
@@ -138,7 +149,7 @@ def test_kfold_cv_api_from_file(tmpdir):
     # generate synthetic data for the test
     input_features = [number_feature(normalization="zscore"), number_feature(normalization="zscore")]
 
-    output_features = [category_feature(vocab_size=3, reduce_input="sum")]
+    output_features = [category_feature(decoder={"vocab_size": 3}, reduce_input="sum")]
 
     generate_data(input_features, output_features, training_data_fp)
 
diff --git a/tests/integration_tests/test_missing_value_strategy.py b/tests/integration_tests/test_missing_value_strategy.py
index 297a0361a5b..8cf061dc074 100644
--- a/tests/integration_tests/test_missing_value_strategy.py
+++ b/tests/integration_tests/test_missing_value_strategy.py
@@ -40,7 +40,9 @@ def test_missing_value_prediction(tmpdir, csv_filename):
     random.seed(1)
     np.random.seed(1)
     input_features = [
-        category_feature(vocab_size=2, reduce_input="sum", preprocessing=dict(missing_value_strategy="fill_with_mode"))
+        category_feature(
+            encoder={"vocab_size": 2}, reduce_input="sum", preprocessing=dict(missing_value_strategy="fill_with_mode")
+        )
     ]
     output_features = [binary_feature()]
 
@@ -72,7 +74,7 @@ def test_missing_values_fill_with_mean(backend, csv_filename, tmpdir):
     input_features = [
         number_feature(**kwargs),
         binary_feature(),
-        category_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
     ]
     output_features = [binary_feature()]
     training_data_csv_path = generate_data(input_features, output_features, data_csv_path)
@@ -91,16 +93,16 @@ def test_missing_values_drop_rows(csv_filename, tmpdir):
     input_features = [
         number_feature(),
         binary_feature(),
-        category_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
     ]
     output_features = [
         binary_feature(**kwargs),
         number_feature(**kwargs),
-        category_feature(vocab_size=3, **kwargs),
-        sequence_feature(vocab_size=3, **kwargs),
-        text_feature(vocab_size=3, **kwargs),
-        set_feature(vocab_size=3, **kwargs),
-        vector_feature(),
+        category_feature(decoder={"vocab_size": 3}, **kwargs),
+        sequence_feature(decoder={"vocab_size": 3}, **kwargs),
+        text_feature(decoder={"vocab_size": 3}, **kwargs),
+        set_feature(decoder={"vocab_size": 3}, **kwargs),
+        vector_feature(**kwargs),
     ]
     backend = LocalTestBackend()
     config = {"input_features": input_features, "output_features": output_features, TRAINER: {"epochs": 2}}
diff --git a/tests/integration_tests/test_mlflow.py b/tests/integration_tests/test_mlflow.py
index cf9586e801a..9e0ad8ebc65 100644
--- a/tests/integration_tests/test_mlflow.py
+++ b/tests/integration_tests/test_mlflow.py
@@ -19,8 +19,8 @@ def test_mlflow_callback(tmpdir):
     batch_size = 8
     num_examples = 32
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     config = {
         "input_features": input_features,
diff --git a/tests/integration_tests/test_model_save_and_load.py b/tests/integration_tests/test_model_save_and_load.py
index a0a98a0e7b1..881172e21f7 100644
--- a/tests/integration_tests/test_model_save_and_load.py
+++ b/tests/integration_tests/test_model_save_and_load.py
@@ -40,27 +40,29 @@ def test_model_save_reload_api(tmpdir, csv_filename, tmp_path):
     input_features = [
         binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
-        sequence_feature(vocab_size=3),
-        text_feature(vocab_size=3, encoder="rnn", cell_type="lstm", num_layers=2, bidirections=True),
+        category_feature(encoder={"vocab_size": 3}),
+        sequence_feature(encoder={"vocab_size": 3}),
+        text_feature(
+            encoder={"vocab_size": 3, "type": "rnn", "cell_type": "lstm", "num_layers": 2, "bidirectional": False}
+        ),
         vector_feature(),
         image_feature(image_dest_folder),
-        audio_feature(audio_dest_folder, encoder="stacked_cnn"),
-        timeseries_feature(encoder="parallel_cnn"),
-        sequence_feature(vocab_size=3, encoder="stacked_parallel_cnn"),
+        audio_feature(audio_dest_folder, encoder={"type": "stacked_cnn"}),
+        timeseries_feature(encoder={"type": "parallel_cnn"}),
+        sequence_feature(encoder={"vocab_size": 3, "type": "stacked_parallel_cnn"}),
         date_feature(),
         h3_feature(),
-        set_feature(vocab_size=3),
-        bag_feature(vocab_size=3),
+        set_feature(encoder={"vocab_size": 3}),
+        bag_feature(encoder={"vocab_size": 3}),
     ]
 
     output_features = [
         binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
-        sequence_feature(vocab_size=3),
-        text_feature(vocab_size=3),
-        set_feature(vocab_size=3),
+        category_feature(decoder={"vocab_size": 3}),
+        sequence_feature(decoder={"vocab_size": 3}),
+        text_feature(decoder={"vocab_size": 3}),
+        set_feature(decoder={"vocab_size": 3}),
         vector_feature(),
     ]
 
@@ -134,8 +136,8 @@ def test_gbm_model_save_reload_api(tmpdir, csv_filename, tmp_path):
     random.seed(1)
     np.random.seed(1)
 
-    input_features = [binary_feature(), number_feature(), category_feature(vocab_size=3)]
-    output_features = [category_feature(vocab_size=3)]
+    input_features = [binary_feature(), number_feature(), category_feature(encoder={"vocab_size": 3})]
+    output_features = [category_feature(decoder={"vocab_size": 3})]
 
     # Generate test data
     data_csv_path = generate_data(input_features, output_features, csv_filename)
diff --git a/tests/integration_tests/test_model_training_options.py b/tests/integration_tests/test_model_training_options.py
index d020de000c1..04ee6dc5f43 100644
--- a/tests/integration_tests/test_model_training_options.py
+++ b/tests/integration_tests/test_model_training_options.py
@@ -401,8 +401,8 @@ def test_regularization(generated_data, tmp_path):
 # test cache checksum function
 def test_cache_checksum(csv_filename, tmp_path):
     # setup for training
-    input_features = [category_feature(vocab_size=5)]
-    output_features = [category_feature(vocab_size=2, top_k=2)]
+    input_features = [category_feature(encoder={"vocab_size": 5})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, top_k=2)]
 
     source_dataset = os.path.join(tmp_path, csv_filename)
     source_dataset = generate_data(input_features, output_features, source_dataset)
@@ -465,7 +465,7 @@ def test_cache_checksum(csv_filename, tmp_path):
 
     # force change in features names (and properties)
     prior_training_timestamp = current_training_timestamp
-    input_features = [category_feature(vocab_size=5), category_feature()]
+    input_features = [category_feature(encoder={"vocab_size": 5}), category_feature()]
     source_dataset = generate_data(input_features, output_features, source_dataset)
     config[INPUT_FEATURES] = input_features
     model = LudwigModel(config, backend=backend)
diff --git a/tests/integration_tests/test_neuropod.py b/tests/integration_tests/test_neuropod.py
index e96b95f072b..800a42ff360 100644
--- a/tests/integration_tests/test_neuropod.py
+++ b/tests/integration_tests/test_neuropod.py
@@ -45,7 +45,7 @@ def test_neuropod_torchscript(csv_filename, tmpdir):
         bin_str_feature,
         # binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
         # TODO: future support
         # sequence_feature(vocab_size=3),
         # text_feature(vocab_size=3),
@@ -62,7 +62,7 @@ def test_neuropod_torchscript(csv_filename, tmpdir):
         bin_str_feature,
         # binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
+        category_feature(decoder={"vocab_size": 3}),
         # TODO: future support
         # sequence_feature(vocab_size=3),
         # text_feature(vocab_size=3),
diff --git a/tests/integration_tests/test_postprocessing.py b/tests/integration_tests/test_postprocessing.py
index dd9248d08b2..8e98c27b064 100644
--- a/tests/integration_tests/test_postprocessing.py
+++ b/tests/integration_tests/test_postprocessing.py
@@ -23,7 +23,7 @@
 import torch
 
 from ludwig.api import LudwigModel
-from ludwig.constants import NAME, TRAINER
+from ludwig.constants import ENCODER, NAME, TRAINER
 from tests.integration_tests.utils import (
     binary_feature,
     category_feature,
@@ -57,7 +57,7 @@ def random_set_logits(*args, num_predict_samples, vocab_size, pct_positive, **kw
 @pytest.mark.parametrize("distinct_values", [(False, True), ("No", "Yes")])
 def test_binary_predictions(tmpdir, backend, distinct_values):
     input_features = [
-        category_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
     ]
 
     feature = binary_feature()
@@ -111,7 +111,7 @@ def test_binary_predictions(tmpdir, backend, distinct_values):
 @pytest.mark.parametrize("distinct_values", [(0.0, 1.0), (0, 1)])
 def test_binary_predictions_with_number_dtype(tmpdir, backend, distinct_values):
     input_features = [
-        category_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
     ]
 
     feature = binary_feature()
@@ -164,7 +164,7 @@ def test_binary_predictions_with_number_dtype(tmpdir, backend, distinct_values):
 def test_set_feature_saving(tmpdir, pct_positive):
     backend = "local"
     input_features = [
-        text_feature(vocab_size=3),
+        text_feature(encoder={"vocab_size": 3}),
     ]
 
     feature = set_feature()
@@ -187,7 +187,7 @@ def test_set_feature_saving(tmpdir, pct_positive):
         partial(
             random_set_logits,
             num_predict_samples=len(data_df),
-            vocab_size=feature["vocab_size"] + 1,  # +1 for UNK
+            vocab_size=feature[ENCODER]["vocab_size"] + 1,  # +1 for UNK
             pct_positive=pct_positive,
         ),
     )
diff --git a/tests/integration_tests/test_preprocessing.py b/tests/integration_tests/test_preprocessing.py
index 81e1831ef57..29b6e634d36 100644
--- a/tests/integration_tests/test_preprocessing.py
+++ b/tests/integration_tests/test_preprocessing.py
@@ -8,7 +8,7 @@
 from PIL import Image
 
 from ludwig.api import LudwigModel
-from ludwig.constants import COLUMN, NAME, PROC_COLUMN, TRAINER
+from ludwig.constants import COLUMN, DECODER, NAME, PROC_COLUMN, TRAINER
 from ludwig.data.concatenate_datasets import concatenate_df
 from tests.integration_tests.utils import (
     audio_feature,
@@ -31,8 +31,8 @@ def test_sample_ratio(backend, tmpdir):
     num_examples = 100
     sample_ratio = 0.25
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
     data_csv = generate_data(
         input_features, output_features, os.path.join(tmpdir, "dataset.csv"), num_examples=num_examples
     )
@@ -61,7 +61,7 @@ def test_strip_whitespace_category(csv_filename, tmpdir):
     data_csv_path = os.path.join(tmpdir, csv_filename)
 
     input_features = [binary_feature()]
-    cat_feat = category_feature(vocab_size=3)
+    cat_feat = category_feature(decoder={"vocab_size": 3})
     output_features = [cat_feat]
     backend = LocalTestBackend()
     config = {"input_features": input_features, "output_features": output_features}
@@ -77,7 +77,7 @@ def test_strip_whitespace_category(csv_filename, tmpdir):
     train_ds, _, _, metadata = ludwig_model.preprocess(dataset=df)
 
     # expect values containing whitespaces to be properly mapped to vocab_size unique values
-    assert len(np.unique(train_ds.dataset[cat_feat[PROC_COLUMN]])) == cat_feat["vocab_size"]
+    assert len(np.unique(train_ds.dataset[cat_feat[PROC_COLUMN]])) == cat_feat[DECODER]["vocab_size"]
 
 
 @pytest.mark.parametrize("backend", ["local", "ray"])
@@ -88,8 +88,8 @@ def test_with_split(backend, csv_filename, tmpdir):
     val_set_size = int(num_examples * 0.1)
     test_set_size = int(num_examples * 0.1)
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
     data_csv = generate_data(
         input_features, output_features, os.path.join(tmpdir, csv_filename), num_examples=num_examples
     )
@@ -122,8 +122,7 @@ def test_dask_known_divisions(feature_fn, csv_filename, tmpdir):
     import dask.dataframe as dd
 
     input_features = [feature_fn(os.path.join(tmpdir, "generated_output"))]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
-
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
     data_csv = generate_data(input_features, output_features, os.path.join(tmpdir, csv_filename), num_examples=100)
     data_df = dd.from_pandas(pd.read_csv(data_csv), npartitions=2)
     assert data_df.known_divisions
@@ -180,7 +179,7 @@ def test_drop_empty_partitions(csv_filename, tmpdir):
 @pytest.mark.parametrize("generate_images_as_numpy", [False, True])
 def test_read_image_from_path(tmpdir, csv_filename, generate_images_as_numpy):
     input_features = [image_feature(os.path.join(tmpdir, "generated_output"), save_as_numpy=generate_images_as_numpy)]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
     data_csv = generate_data(
         input_features, output_features, os.path.join(tmpdir, csv_filename), num_examples=NUM_EXAMPLES
     )
@@ -200,7 +199,7 @@ def test_read_image_from_path(tmpdir, csv_filename, generate_images_as_numpy):
 
 def test_read_image_from_numpy_array(tmpdir, csv_filename):
     input_features = [image_feature(os.path.join(tmpdir, "generated_output"))]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     config = {
         "input_features": input_features,
@@ -267,8 +266,8 @@ def random_string():
 def test_presplit_override(format, tmpdir):
     """Tests that provising a pre-split file or dataframe overrides the user's split config."""
     num_feat = number_feature(normalization=None)
-    input_features = [num_feat, sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [num_feat, sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     data_csv = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"), num_examples=25)
     data_df = pd.read_csv(data_csv)
diff --git a/tests/integration_tests/test_ray.py b/tests/integration_tests/test_ray.py
index 715ab7f333b..945d942ea29 100644
--- a/tests/integration_tests/test_ray.py
+++ b/tests/integration_tests/test_ray.py
@@ -225,10 +225,10 @@ def test_ray_read_binary_files(tmpdir, df_engine):
 @pytest.mark.distributed
 def test_ray_save_processed_input(dataset_type):
     input_features = [
-        category_feature(vocab_size=2, reduce_input="sum"),
+        category_feature(encoder={"vocab_size": 2}, reduce_input="sum"),
     ]
     output_features = [
-        category_feature(vocab_size=5),  # Regression test for #1991 requires multi-class predictions.
+        category_feature(decoder={"vocab_size": 5}),  # Regression test for #1991 requires multi-class predictions.
     ]
     run_test_with_features(
         input_features,
@@ -244,8 +244,8 @@ def test_ray_save_processed_input(dataset_type):
 @pytest.mark.distributed
 def test_ray_tabular(df_engine):
     input_features = [
-        sequence_feature(reduce_output="sum"),
-        category_feature(vocab_size=2, reduce_input="sum"),
+        sequence_feature(encoder={"reduce_output": "sum"}),
+        category_feature(encoder={"vocab_size": 2}, reduce_input="sum"),
         number_feature(normalization="zscore"),
         set_feature(),
         binary_feature(),
@@ -273,7 +273,7 @@ def test_ray_text():
         text_feature(),
     ]
     output_features = [
-        text_feature(reduce_input=None, decoder="tagger"),
+        text_feature(reduce_input=None, decoder={"type": "tagger"}),
     ]
     run_test_with_features(input_features, output_features)
 
@@ -281,8 +281,12 @@ def test_ray_text():
 @pytest.mark.skip(reason="TODO torch")
 @pytest.mark.distributed
 def test_ray_sequence():
-    input_features = [sequence_feature(max_len=10, encoder="rnn", cell_type="lstm", reduce_output=None)]
-    output_features = [sequence_feature(max_len=10, decoder="tagger", attention=False, reduce_input=None)]
+    input_features = [
+        sequence_feature(encoder={"max_len": 10, "type": "rnn", "cell_type": "lstm", "reduce_output": None})
+    ]
+    output_features = [
+        sequence_feature(decoder={"max_len": 10, "type": "tagger", "attention": False}, reduce_input=None)
+    ]
     run_test_with_features(input_features, output_features)
 
 
@@ -341,10 +345,8 @@ def test_ray_image_modin(tmpdir):
     input_features = [
         image_feature(
             folder=image_dest_folder,
-            encoder="resnet",
+            encoder={"type": "resnet", "output_size": 16, "num_filters": 8},
             preprocessing={"in_memory": True, "height": 12, "width": 12, "num_channels": 3, "num_processes": 5},
-            output_size=16,
-            num_filters=8,
         ),
     ]
     output_features = [binary_feature()]
@@ -391,7 +393,7 @@ def test_ray_split():
         set_feature(),
         binary_feature(),
     ]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
     run_test_with_features(
         input_features,
         output_features,
@@ -428,10 +430,8 @@ def test_ray_lazy_load_image_error(tmpdir):
     input_features = [
         image_feature(
             folder=image_dest_folder,
-            encoder="resnet",
+            encoder={"type": "resnet", "output_size": 16, "num_filters": 8},
             preprocessing={"in_memory": False, "height": 12, "width": 12, "num_channels": 3, "num_processes": 5},
-            output_size=16,
-            num_filters=8,
         ),
     ]
     output_features = [binary_feature()]
@@ -443,7 +443,7 @@ def test_ray_lazy_load_image_error(tmpdir):
 @pytest.mark.distributed
 def test_train_gpu_load_cpu():
     input_features = [
-        category_feature(vocab_size=2, reduce_input="sum"),
+        category_feature(encoder={"vocab_size": 2}, reduce_input="sum"),
         number_feature(normalization="zscore"),
     ]
     output_features = [
@@ -513,7 +513,7 @@ def test_tune_batch_size_lr(tmpdir):
                 set_feature(),
                 binary_feature(),
             ],
-            "output_features": [category_feature(vocab_size=2, reduce_input="sum")],
+            "output_features": [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")],
             "combiner": {"type": "concat", "output_size": 14},
             TRAINER: {"epochs": 2, "batch_size": "auto", "learning_rate": "auto"},
         }
@@ -532,7 +532,7 @@ def test_tune_batch_size_lr(tmpdir):
 def test_ray_progress_bar():
     # This is a simple test that is just meant to make sure that the progress bar isn't breaking
     input_features = [
-        sequence_feature(reduce_output="sum"),
+        sequence_feature(encoder={"reduce_output": "sum"}),
     ]
     output_features = [
         binary_feature(bool2str=["No", "Yes"]),
@@ -554,6 +554,6 @@ def test_ray_calibration(calibration):
     ]
     output_features = [
         binary_feature(calibration=calibration),
-        category_feature(vocab_size=3, calibration=calibration),
+        category_feature(decoder={"vocab_size": 3}, calibration=calibration),
     ]
     run_test_with_features(input_features, output_features)
diff --git a/tests/integration_tests/test_regularizers.py b/tests/integration_tests/test_regularizers.py
index 899bc805fe2..cc040ceed5d 100644
--- a/tests/integration_tests/test_regularizers.py
+++ b/tests/integration_tests/test_regularizers.py
@@ -32,17 +32,17 @@
     "input_features,output_features",
     [
         (
-            [number_feature(num_layers=2, encoder="dense", preprocessing={"normalization": "zscore"})],
+            [number_feature(encoder={"num_layers": 2, "type": "dense"}, preprocessing={"normalization": "zscore"})],
             [number_feature()],
         ),
-        ([image_feature(IMAGE_DIR, encoder="stacked_cnn")], [number_feature()]),
-        ([image_feature(IMAGE_DIR, encoder="resnet")], [category_feature()]),
+        ([image_feature(IMAGE_DIR, encoder={"type": "stacked_cnn"})], [number_feature()]),
+        ([image_feature(IMAGE_DIR, encoder={"type": "resnet"})], [category_feature()]),
         (
-            [category_feature(representation="dense")],
-            [number_feature(decoder="regressor", loss={"type": "mean_squared_error"}, num_fc_layers=5)],
+            [category_feature(encoder={"representation": "dense"})],
+            [number_feature(decoder={"type": "regressor", "num_fc_layers": 5}, loss={"type": "mean_squared_error"})],
         ),
         ([date_feature()], [binary_feature()]),
-        ([sequence_feature(encoder="parallel_cnn", cell_type="gru")], [binary_feature()]),
+        ([sequence_feature(encoder={"type": "parallel_cnn", "cell_type": "gru"})], [binary_feature()]),
         ([set_feature()], [set_feature()]),
     ],
 )
diff --git a/tests/integration_tests/test_remote.py b/tests/integration_tests/test_remote.py
index c9054794b9e..08333da6ae9 100644
--- a/tests/integration_tests/test_remote.py
+++ b/tests/integration_tests/test_remote.py
@@ -14,8 +14,8 @@
 def test_remote_training_set(tmpdir, fs_protocol):
     output_directory = f"{fs_protocol}://{tmpdir}"
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     csv_filename = os.path.join(tmpdir, "training.csv")
     data_csv = generate_data(input_features, output_features, csv_filename)
diff --git a/tests/integration_tests/test_sequence_features.py b/tests/integration_tests/test_sequence_features.py
index 98e172d1131..96dd19608f7 100644
--- a/tests/integration_tests/test_sequence_features.py
+++ b/tests/integration_tests/test_sequence_features.py
@@ -7,7 +7,7 @@
 import torch
 
 from ludwig.api import LudwigModel
-from ludwig.constants import LOGITS
+from ludwig.constants import DECODER, LOGITS
 from ludwig.data.dataset_synthesizer import build_synthetic_dataset
 from ludwig.data.preprocessing import preprocess_for_training
 from ludwig.features.feature_registries import update_config_with_metadata
@@ -31,20 +31,24 @@
 def generate_sequence_training_data():
     input_features = [
         sequence_feature(
-            vocab_size=TEST_VOCAB_SIZE,
-            embedding_size=TEST_EMBEDDING_SIZE,
-            state_size=TEST_STATE_SIZE,
-            hidden_size=TEST_HIDDEN_SIZE,
-            num_filters=TEST_NUM_FILTERS,
-            min_len=5,
-            max_len=10,
-            encoder="rnn",
-            cell_type="lstm",
+            encoder={
+                "vocab_size": TEST_VOCAB_SIZE,
+                "embedding_size": TEST_EMBEDDING_SIZE,
+                "state_size": TEST_STATE_SIZE,
+                "hidden_size": TEST_HIDDEN_SIZE,
+                "num_filters": TEST_NUM_FILTERS,
+                "min_len": 5,
+                "max_len": 10,
+                "type": "rnn",
+                "cell_type": "lstm",
+            }
         )
     ]
 
     output_features = [
-        sequence_feature(min_len=5, max_len=10, decoder="generator", cell_type="lstm", attention="bahdanau")
+        sequence_feature(
+            decoder={"type": "generator", "min_len": 5, "max_len": 10, "cell_type": "lstm", "attention": "bahdanau"}
+        )
     ]
 
     # generate synthetic data set testing
@@ -110,7 +114,7 @@ def test_sequence_decoders(
     input_features = generate_sequence_training_data[1]
     output_features = generate_sequence_training_data[2]
     output_feature_name = output_features[0]["name"]
-    output_features[0]["cell_type"] = dec_cell_type
+    output_features[0][DECODER]["cell_type"] = dec_cell_type
 
     with setup_model_scaffolding(raw_df, input_features, output_features) as (model, _):
 
@@ -135,8 +139,8 @@ def test_sequence_decoders(
 
         # gather expected components of the shape
         batch_size = combiner_outputs["hidden"].shape[0]
-        seq_size = output_features[0]["max_len"] + 2  # For start and stop symbols.
-        vocab_size = model.config["output_features"][0]["vocab_size"]
+        seq_size = output_features[0][DECODER]["max_len"] + 2  # For start and stop symbols.
+        vocab_size = model.config["output_features"][0][DECODER]["vocab_size"]
 
         # confirm shape and format of decoder output
         assert list(decoder_out[LOGITS].size()) == [batch_size, seq_size, vocab_size]
@@ -148,8 +152,12 @@ def test_sequence_decoders(
 @pytest.mark.parametrize("enc_encoder", ["embed", "rnn"])
 def test_sequence_generator(enc_encoder, enc_cell_type, dec_cell_type, csv_filename):
     # Define input and output features
-    input_features = [sequence_feature(min_len=5, max_len=10, encoder=enc_encoder, cell_type=enc_cell_type)]
-    output_features = [sequence_feature(min_len=5, max_len=10, decoder="generator", cell_type=dec_cell_type)]
+    input_features = [
+        sequence_feature(encoder={"type": enc_encoder, "min_len": 5, "max_len": 10, "cell_type": enc_cell_type})
+    ]
+    output_features = [
+        sequence_feature(decoder={"type": "generator", "min_len": 5, "max_len": 10, "cell_type": dec_cell_type})
+    ]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
diff --git a/tests/integration_tests/test_server.py b/tests/integration_tests/test_server.py
index 97cfe06afbf..700d2abb8b5 100644
--- a/tests/integration_tests/test_server.py
+++ b/tests/integration_tests/test_server.py
@@ -125,14 +125,13 @@ def test_server_integration_with_images(tmpdir):
     input_features = [
         image_feature(
             folder=image_dest_folder,
+            encoder={"output_size": 16, "num_filters": 8},
             preprocessing={"in_memory": True, "height": 8, "width": 8, "num_channels": 3},
-            output_size=16,
-            num_filters=8,
         ),
-        text_feature(encoder="embed", min_len=1),
+        text_feature(encoder={"type": "embed", "min_len": 1}),
         number_feature(normalization="zscore"),
     ]
-    output_features = [category_feature(vocab_size=4), number_feature()]
+    output_features = [category_feature(decoder={"vocab_size": 4}), number_feature()]
 
     np.random.seed(123)  # reproducible synthetic data
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
@@ -191,10 +190,10 @@ def test_server_integration_with_audio(single_record, tmpdir):
         audio_feature(
             folder=audio_dest_folder,
         ),
-        text_feature(encoder="embed", min_len=1),
+        text_feature(encoder={"type": "embed", "min_len": 1}),
         number_feature(normalization="zscore"),
     ]
-    output_features = [category_feature(vocab_size=4), number_feature()]
+    output_features = [category_feature(decoder={"vocab_size": 4}), number_feature()]
 
     rel_path = generate_data(input_features, output_features, os.path.join(tmpdir, "dataset.csv"))
 
diff --git a/tests/integration_tests/test_simple_features.py b/tests/integration_tests/test_simple_features.py
index 05e001aad7a..07c5bbfcd68 100644
--- a/tests/integration_tests/test_simple_features.py
+++ b/tests/integration_tests/test_simple_features.py
@@ -69,11 +69,20 @@ def test_feature(input_test_feature, output_test_feature, output_loss_parameter,
     "input_test_feature, output_test_feature",
     [
         ([category_feature()], [binary_feature(), binary_feature()]),
-        ([category_feature()], [category_feature(vocab_size=5), category_feature(vocab_size=7)]),
+        (
+            [category_feature()],
+            [category_feature(decoder={"vocab_size": 5}), category_feature(decoder={"vocab_size": 7})],
+        ),
         ([category_feature()], [number_feature(), number_feature()]),
-        ([category_feature()], [sequence_feature(vocab_size=5), sequence_feature(vocab_size=7)]),
-        ([set_feature(vocab_size=5)], [set_feature(vocab_size=5), set_feature(vocab_size=7)]),
-        ([category_feature()], [text_feature(vocab_size=5), text_feature(vocab_size=7)]),
+        (
+            [category_feature()],
+            [sequence_feature(decoder={"vocab_size": 5}), sequence_feature(decoder={"vocab_size": 7})],
+        ),
+        (
+            [set_feature(encoder={"vocab_size": 5})],
+            [set_feature(decoder={"vocab_size": 5}), set_feature(decoder={"vocab_size": 7})],
+        ),
+        ([category_feature()], [text_feature(decoder={"vocab_size": 5}), text_feature(decoder={"vocab_size": 7})]),
         ([category_feature()], [vector_feature(), vector_feature()]),
         ([vector_feature()], [vector_feature(), vector_feature()]),
         ([bag_feature()], [vector_feature(), vector_feature()]),
diff --git a/tests/integration_tests/test_timeseries_feature.py b/tests/integration_tests/test_timeseries_feature.py
index 6cb31c3e111..c081511c612 100644
--- a/tests/integration_tests/test_timeseries_feature.py
+++ b/tests/integration_tests/test_timeseries_feature.py
@@ -18,11 +18,13 @@ def test_timeseries_feature(enc_encoder):
 
     # generate feature config
     timeseries_feature_config = timeseries_feature(
-        encoder=enc_encoder,
-        max_len=SEQ_SIZE,
-        fc_layers=[{"output_size": DEFAULT_OUTPUT_SIZE}],
-        # simulated parameters determined by pre-processing
-        max_sequence_length=SEQ_SIZE,
+        encoder={
+            "type": enc_encoder,
+            "max_len": SEQ_SIZE,
+            "fc_layers": [{"output_size": DEFAULT_OUTPUT_SIZE}],
+            # simulated parameters determined by pre-processing
+            "max_sequence_length": SEQ_SIZE,
+        }
     )
 
     # instantiate input feature object
diff --git a/tests/integration_tests/test_torchscript.py b/tests/integration_tests/test_torchscript.py
index e4e76350dbf..b451b1d6b36 100644
--- a/tests/integration_tests/test_torchscript.py
+++ b/tests/integration_tests/test_torchscript.py
@@ -65,15 +65,15 @@ def test_torchscript(tmpdir, csv_filename, should_load_model, model_type):
     input_features = [
         binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
     ]
     if model_type == "ecd":
         image_dest_folder = os.path.join(tmpdir, "generated_images")
         audio_dest_folder = os.path.join(tmpdir, "generated_audio")
         input_features.extend(
             [
-                sequence_feature(vocab_size=3),
-                text_feature(vocab_size=3),
+                sequence_feature(encoder={"vocab_size": 3}),
+                text_feature(encoder={"vocab_size": 3}),
                 vector_feature(),
                 image_feature(image_dest_folder),
                 audio_feature(audio_dest_folder),
@@ -81,23 +81,23 @@ def test_torchscript(tmpdir, csv_filename, should_load_model, model_type):
                 date_feature(),
                 date_feature(),
                 h3_feature(),
-                set_feature(vocab_size=3),
-                bag_feature(vocab_size=3),
+                set_feature(encoder={"vocab_size": 3}),
+                bag_feature(encoder={"vocab_size": 3}),
             ]
         )
 
     output_features = [
-        category_feature(vocab_size=3),
+        category_feature(decoder={"vocab_size": 3}),
     ]
     if model_type == "ecd":
         output_features.extend(
             [
                 binary_feature(),
                 number_feature(),
-                set_feature(vocab_size=3),
+                set_feature(decoder={"vocab_size": 3}),
                 vector_feature(),
-                sequence_feature(vocab_size=3),
-                text_feature(vocab_size=3),
+                sequence_feature(decoder={"vocab_size": 3}),
+                text_feature(decoder={"vocab_size": 3}),
             ]
         )
 
@@ -227,9 +227,9 @@ def test_torchscript_e2e_tabular(csv_filename, tmpdir):
         bin_str_feature,
         binary_feature(),
         *transformed_number_features,
-        category_feature(vocab_size=3),
-        bag_feature(vocab_size=3),
-        set_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
+        bag_feature(encoder={"vocab_size": 3}),
+        set_feature(encoder={"vocab_size": 3}),
         vector_feature(),
         # TODO: future support
         # date_feature(),
@@ -239,11 +239,11 @@ def test_torchscript_e2e_tabular(csv_filename, tmpdir):
         bin_str_feature,
         binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
-        set_feature(vocab_size=3),
+        category_feature(decoder={"vocab_size": 3}),
+        set_feature(decoder={"vocab_size": 3}),
         vector_feature(),
-        sequence_feature(vocab_size=3),
-        text_feature(vocab_size=3),
+        sequence_feature(decoder={"vocab_size": 3}),
+        text_feature(decoder={"vocab_size": 3}),
     ]
     backend = LocalTestBackend()
     config = {"input_features": input_features, "output_features": output_features, TRAINER: {"epochs": 2}}
@@ -284,14 +284,14 @@ def test_torchscript_e2e_tabnet_combiner(csv_filename, tmpdir):
     input_features = [
         binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
-        bag_feature(vocab_size=3),
-        set_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
+        bag_feature(encoder={"vocab_size": 3}),
+        set_feature(encoder={"vocab_size": 3}),
     ]
     output_features = [
         binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
+        category_feature(decoder={"vocab_size": 3}),
     ]
     backend = LocalTestBackend()
     config = {
@@ -350,11 +350,11 @@ def test_torchscript_e2e_image(tmpdir, csv_filename):
 def test_torchscript_e2e_text(tmpdir, csv_filename):
     data_csv_path = os.path.join(tmpdir, csv_filename)
     input_features = [
-        text_feature(vocab_size=3, preprocessing={"tokenizer": tokenizer})
+        text_feature(encoder={"vocab_size": 3}, preprocessing={"tokenizer": tokenizer})
         for tokenizer in TORCHSCRIPT_COMPATIBLE_TOKENIZERS
     ]
     output_features = [
-        text_feature(vocab_size=3),
+        text_feature(decoder={"vocab_size": 3}),
     ]
     backend = LocalTestBackend()
     config = {"input_features": input_features, "output_features": output_features, TRAINER: {"epochs": 2}}
@@ -369,9 +369,9 @@ def test_torchscript_e2e_text(tmpdir, csv_filename):
 )
 def test_torchscript_e2e_text_hf_tokenizer(tmpdir, csv_filename):
     data_csv_path = os.path.join(tmpdir, csv_filename)
-    input_features = [text_feature(vocab_size=3, encoder="bert")]
+    input_features = [text_feature(encoder={"vocab_size": 3, "type": "bert"})]
     output_features = [
-        text_feature(vocab_size=3),
+        text_feature(decoder={"vocab_size": 3}),
     ]
     backend = LocalTestBackend()
     config = {"input_features": input_features, "output_features": output_features, TRAINER: {"epochs": 2}}
@@ -383,10 +383,10 @@ def test_torchscript_e2e_text_hf_tokenizer(tmpdir, csv_filename):
 def test_torchscript_e2e_sequence(tmpdir, csv_filename):
     data_csv_path = os.path.join(tmpdir, csv_filename)
     input_features = [
-        sequence_feature(vocab_size=3, preprocessing={"tokenizer": "space"}),
+        sequence_feature(encoder={"vocab_size": 3}, preprocessing={"tokenizer": "space"}),
     ]
     output_features = [
-        sequence_feature(vocab_size=3),
+        sequence_feature(decoder={"vocab_size": 3}),
     ]
     backend = LocalTestBackend()
     config = {"input_features": input_features, "output_features": output_features, TRAINER: {"epochs": 2}}
@@ -502,7 +502,7 @@ def test_torchscript_preproc_timeseries_alternative_type(tmpdir, csv_filename, p
             "timeseries_length_limit": 4,
             "fill_value": "1.0",
         },
-        max_len=7,
+        encoder={"max_len": 7},
     )
     input_features = [
         feature,
@@ -553,11 +553,11 @@ def transform_timeseries_from_str_list_to_tensor_list(timeseries_list):
     [
         number_feature(),
         binary_feature(),
-        category_feature(vocab_size=3),
-        bag_feature(vocab_size=3),
-        set_feature(vocab_size=3),
-        text_feature(vocab_size=3),
-        sequence_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
+        bag_feature(encoder={"vocab_size": 3}),
+        set_feature(encoder={"vocab_size": 3}),
+        text_feature(encoder={"vocab_size": 3}),
+        sequence_feature(encoder={"vocab_size": 3}),
         timeseries_feature(),
         h3_feature(),
         # TODO: future support
@@ -618,12 +618,12 @@ def test_torchscript_preproc_with_nans(tmpdir, csv_filename, feature):
         date_feature,
         # TODO: future support
         # binary_feature(),                # Torchscript takes List[str] as input, so currently CPU only
-        # category_feature(vocab_size=3),  # Torchscript takes List[str] as input, so currently CPU only
-        # set_feature(vocab_size=3),       # Torchscript takes List[str] as input, so currently CPU only
-        # sequence_feature(vocab_size=3),  # Torchscript takes List[str] as input, so currently CPU only
-        # text_feature(vocab_size=3),      # Torchscript takes List[str] as input, so currently CPU only
+        # category_feature(encoder={"vocab_size": 3}),  # Torchscript takes List[str] as input, so currently CPU only
+        # set_feature(encoder={"vocab_size": 3}),       # Torchscript takes List[str] as input, so currently CPU only
+        # sequence_feature(encoder={"vocab_size": 3}),  # Torchscript takes List[str] as input, so currently CPU only
+        # text_feature(encoder={"vocab_size": 3}),      # Torchscript takes List[str] as input, so currently CPU only
         # vector_feature(),                # Torchscript takes List[str] as input, so currently CPU only
-        # bag_feature(vocab_size=3),       # Torchscript takes List[str] as input, so currently CPU only
+        # bag_feature(encoder={"vocab_size": 3}),       # Torchscript takes List[str] as input, so currently CPU only
         # timeseries_feature(),            # Torchscript takes List[str] as input, so currently CPU only
     ],
 )
diff --git a/tests/integration_tests/test_trainer.py b/tests/integration_tests/test_trainer.py
index faf5cf4249f..c5cf385adc7 100644
--- a/tests/integration_tests/test_trainer.py
+++ b/tests/integration_tests/test_trainer.py
@@ -56,8 +56,8 @@ def ray_test_cluster():
 
 @pytest.mark.parametrize("eval_batch_size", ["auto", None, 128])
 def test_tune_batch_size_and_lr(tmpdir, eval_batch_size):
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     csv_filename = os.path.join(tmpdir, "training.csv")
     data_csv = generate_data(input_features, output_features, csv_filename)
@@ -124,8 +124,8 @@ def test_scale_lr(learning_rate_scaling, expected_lr, tmpdir, ray_test_cluster):
 
     outdir = os.path.join(tmpdir, "output")
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     csv_filename = os.path.join(tmpdir, "training.csv")
     data_csv = generate_data(input_features, output_features, csv_filename)
@@ -146,8 +146,8 @@ def test_scale_lr(learning_rate_scaling, expected_lr, tmpdir, ray_test_cluster):
 
 
 def test_changing_parameters_on_plateau(tmpdir):
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     csv_filename = os.path.join(tmpdir, "training.csv")
     data_csv = generate_data(input_features, output_features, csv_filename)
diff --git a/tests/integration_tests/test_triton.py b/tests/integration_tests/test_triton.py
index 6a217ed14eb..397d4cfdfa0 100644
--- a/tests/integration_tests/test_triton.py
+++ b/tests/integration_tests/test_triton.py
@@ -39,26 +39,26 @@ def test_triton_torchscript(csv_filename, tmpdir):
     input_features = [
         binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
+        category_feature(encoder={"vocab_size": 3}),
         # TODO: future support
-        # sequence_feature(vocab_size=3),
-        # text_feature(vocab_size=3),
+        # sequence_feature(encoder={"vocab_size": 3}),
+        # text_feature(encoder={"vocab_size": 3}),
         # vector_feature(),
         # timeseries_feature(),
         # date_feature(),
         # h3_feature(),
-        # set_feature(vocab_size=3),
-        # bag_feature(vocab_size=3),
+        # set_feature(encoder={"vocab_size": 3}),
+        # bag_feature(encoder={"vocab_size": 3}),
         # image_feature(image_dest_folder),
         # audio_feature(audio_dest_folder),
     ]
     output_features = [
         binary_feature(),
         number_feature(),
-        category_feature(vocab_size=3),
-        sequence_feature(vocab_size=3),
-        text_feature(vocab_size=3),
-        set_feature(vocab_size=3),
+        category_feature(decoder={"vocab_size": 3}),
+        sequence_feature(decoder={"vocab_size": 3}),
+        text_feature(decoder={"vocab_size": 3}),
+        set_feature(decoder={"vocab_size": 3}),
         vector_feature(),
     ]
     backend = LocalTestBackend()
diff --git a/tests/integration_tests/test_visualization.py b/tests/integration_tests/test_visualization.py
index 870838c62c5..d609a8f1508 100644
--- a/tests/integration_tests/test_visualization.py
+++ b/tests/integration_tests/test_visualization.py
@@ -27,7 +27,7 @@
 import numpy as np
 import pytest
 
-from ludwig.constants import TRAINER
+from ludwig.constants import ENCODER, TRAINER, TYPE
 from ludwig.experiment import experiment_cli
 from ludwig.globals import DESCRIPTION_FILE_NAME, PREDICTIONS_PARQUET_FILE_NAME, TEST_STATISTICS_FILE_NAME
 from ludwig.utils.data_utils import get_split_path
@@ -94,12 +94,12 @@ def test_visualization_learning_curves_output_saved(csv_filename):
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [text_feature(encoder="parallel_cnn")]
+    input_features = [text_feature(encoder={"type": "parallel_cnn"})]
     output_features = [category_feature()]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
-    input_features[0]["encoder"] = "parallel_cnn"
+    input_features[0][ENCODER][TYPE] = "parallel_cnn"
     exp_dir_name = run_experiment_with_visualization(input_features, output_features, dataset=rel_path)
 
     vis_output_pattern_pdf = os.path.join(exp_dir_name, "*.pdf")
@@ -137,12 +137,12 @@ def test_visualization_confusion_matrix_output_saved(csv_filename):
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [text_feature(encoder="parallel_cnn")]
+    input_features = [text_feature(encoder={"type": "parallel_cnn"})]
     output_features = [category_feature()]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
-    input_features[0]["encoder"] = "parallel_cnn"
+    input_features[0][ENCODER][TYPE] = "parallel_cnn"
     exp_dir_name = run_experiment_with_visualization(input_features, output_features, dataset=rel_path)
     vis_output_pattern_pdf = os.path.join(exp_dir_name, "*.pdf")
     vis_output_pattern_png = os.path.join(exp_dir_name, "*.png")
@@ -183,12 +183,12 @@ def test_visualization_compare_performance_output_saved(csv_filename):
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [text_feature(encoder="parallel_cnn")]
+    input_features = [text_feature(encoder={"type": "parallel_cnn"})]
     output_features = [category_feature()]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
-    input_features[0]["encoder"] = "parallel_cnn"
+    input_features[0][ENCODER][TYPE] = "parallel_cnn"
     exp_dir_name = run_experiment_with_visualization(input_features, output_features, dataset=rel_path)
     vis_output_pattern_pdf = os.path.join(exp_dir_name, "*.pdf")
     vis_output_pattern_png = os.path.join(exp_dir_name, "*.png")
@@ -229,8 +229,8 @@ def test_visualization_compare_classifiers_from_prob_csv_output_saved(csv_filena
     :param csv_filename: csv fixture from tests.fixtures.filenames.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -286,8 +286,8 @@ def test_visualization_compare_classifiers_from_prob_npy_output_saved(csv_filena
     :param csv_filename: csv fixture from tests.fixtures.filenames.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -343,8 +343,8 @@ def test_visualization_compare_classifiers_from_pred_npy_output_saved(csv_filena
     :param csv_filename: csv fixture from tests.fixtures.filenames.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -400,8 +400,8 @@ def test_visualization_compare_classifiers_from_pred_csv_output_saved(csv_filena
     :param csv_filename: csv fixture from tests.fixtures.filenames.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -456,8 +456,8 @@ def test_visualization_compare_classifiers_subset_output_saved(csv_filename):
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -509,8 +509,8 @@ def test_visualization_compare_classifiers_subset_output_saved(csv_filename):
 
 def test_visualization_compare_classifiers_changing_k_output_pdf(csv_filename):
     """It should be possible to save figures as pdf in the specified directory."""
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -566,8 +566,8 @@ def test_visualization_compare_classifiers_multiclass_multimetric_output_saved(c
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -614,8 +614,8 @@ def test_visualization_compare_classifiers_predictions_npy_output_saved(csv_file
     :param csv_filename: csv fixture from tests.fixtures.filenames.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -670,8 +670,8 @@ def test_visualization_compare_classifiers_predictions_csv_output_saved(csv_file
     :param csv_filename: csv fixture from tests.fixtures.filenames.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -725,8 +725,8 @@ def test_visualization_cmp_classifiers_predictions_distribution_output_saved(csv
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -780,8 +780,8 @@ def test_visualization_cconfidence_thresholding_output_saved(csv_filename):
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -835,8 +835,8 @@ def test_visualization_confidence_thresholding_data_vs_acc_output_saved(csv_file
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -890,8 +890,8 @@ def test_visualization_confidence_thresholding_data_vs_acc_subset_output_saved(c
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -947,8 +947,8 @@ def test_vis_confidence_thresholding_data_vs_acc_subset_per_class_output_saved(c
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=5, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 5}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -1007,19 +1007,19 @@ def test_vis_confidence_thresholding_2thresholds_2d_output_saved(csv_filename):
     :return: None
     """
     input_features = [
-        text_feature(vocab_size=10, min_len=1, encoder="stacked_cnn"),
+        text_feature(encoder={"vocab_size": 10, "min_len": 1, "type": "stacked_cnn"}),
         number_feature(),
-        category_feature(vocab_size=10, embedding_size=5),
+        category_feature(encoder={"vocab_size": 10, "embedding_size": 5}),
         set_feature(),
-        sequence_feature(vocab_size=10, max_len=10, encoder="embed"),
+        sequence_feature(encoder={"vocab_size": 10, "max_len": 10, "type": "embed"}),
     ]
     output_features = [
-        category_feature(vocab_size=2, reduce_input="sum"),
-        category_feature(vocab_size=2, reduce_input="sum"),
+        category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
+        category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
     ]
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
-    input_features[0]["encoder"] = "parallel_cnn"
+    input_features[0][ENCODER][TYPE] = "parallel_cnn"
     exp_dir_name = run_experiment_with_visualization(input_features, output_features, dataset=rel_path)
     vis_output_pattern_pdf = os.path.join(exp_dir_name, "*.pdf")
     vis_output_pattern_png = os.path.join(exp_dir_name, "*.png")
@@ -1073,19 +1073,19 @@ def test_vis_confidence_thresholding_2thresholds_3d_output_saved(csv_filename):
     :return: None
     """
     input_features = [
-        text_feature(vocab_size=10, min_len=1, encoder="stacked_cnn"),
+        text_feature(encoder={"vocab_size": 10, "min_len": 1, "type": "stacked_cnn"}),
         number_feature(),
-        category_feature(vocab_size=10, embedding_size=5),
+        category_feature(encoder={"vocab_size": 10, "embedding_size": 5}),
         set_feature(),
-        sequence_feature(vocab_size=10, max_len=10, encoder="embed"),
+        sequence_feature(encoder={"vocab_size": 10, "max_len": 10, "type": "embed"}),
     ]
     output_features = [
-        category_feature(vocab_size=2, reduce_input="sum"),
-        category_feature(vocab_size=2, reduce_input="sum"),
+        category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
+        category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
     ]
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
-    input_features[0]["encoder"] = "parallel_cnn"
+    input_features[0][ENCODER][TYPE] = "parallel_cnn"
     exp_dir_name = run_experiment_with_visualization(input_features, output_features, dataset=rel_path)
     vis_output_pattern_pdf = os.path.join(exp_dir_name, "*.pdf")
     vis_output_pattern_png = os.path.join(exp_dir_name, "*.png")
@@ -1138,21 +1138,21 @@ def test_visualization_binary_threshold_vs_metric_output_saved(csv_filename, bin
     :return: None
     """
     input_features = [
-        text_feature(vocab_size=10, min_len=1, encoder="stacked_cnn"),
+        text_feature(encoder={"vocab_size": 10, "min_len": 1, "type": "stacked_cnn"}),
         number_feature(),
-        category_feature(vocab_size=10, embedding_size=5),
+        category_feature(encoder={"vocab_size": 10, "embedding_size": 5}),
         set_feature(),
-        sequence_feature(vocab_size=10, max_len=10, encoder="embed"),
+        sequence_feature(encoder={"vocab_size": 10, "max_len": 10, "type": "embed"}),
     ]
     if binary_output_type:
         output_features = [binary_feature()]
     else:
-        output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+        output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     random.seed(1919)
     rel_path = generate_data(input_features, output_features, csv_filename)
-    input_features[0]["encoder"] = "parallel_cnn"
+    input_features[0][ENCODER][TYPE] = "parallel_cnn"
     exp_dir_name = run_experiment_with_visualization(input_features, output_features, dataset=rel_path)
     vis_output_pattern_pdf = os.path.join(exp_dir_name, "*.pdf")
     vis_output_pattern_png = os.path.join(exp_dir_name, "*.png")
@@ -1211,11 +1211,11 @@ def test_visualization_roc_curves_output_saved(csv_filename, binary_output_type)
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
     if binary_output_type:
         output_features = [binary_feature()]
     else:
-        output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+        output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -1317,8 +1317,8 @@ def test_visualization_calibration_1_vs_all_output_saved(csv_filename):
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -1376,8 +1376,8 @@ def test_visualization_calibration_multiclass_output_saved(csv_filename):
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -1431,8 +1431,8 @@ def test_visualization_frequency_vs_f1_output_saved(csv_filename):
     :param csv_filename: csv fixture from tests.conftest.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
@@ -1511,8 +1511,8 @@ def test_load_ground_truth_split_from_file(csv_filename):
     :param csv_filename: csv fixture from tests.fixtures.filenames.csv_filename
     :return: None
     """
-    input_features = [category_feature(vocab_size=10)]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [category_feature(encoder={"vocab_size": 10})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     # Generate test data
     rel_path = generate_data(input_features, output_features, csv_filename)
diff --git a/tests/integration_tests/test_visualization_api.py b/tests/integration_tests/test_visualization_api.py
index 2cf1bd17ae3..2244da51fa2 100644
--- a/tests/integration_tests/test_visualization_api.py
+++ b/tests/integration_tests/test_visualization_api.py
@@ -22,7 +22,18 @@
 
 from ludwig import visualize
 from ludwig.api import LudwigModel
-from ludwig.constants import NAME, PREDICTIONS, PROBABILITIES, PROBABILITY, TEST, TRAINER, TRAINING, VALIDATION
+from ludwig.constants import (
+    ENCODER,
+    NAME,
+    PREDICTIONS,
+    PROBABILITIES,
+    PROBABILITY,
+    TEST,
+    TRAINER,
+    TRAINING,
+    TYPE,
+    VALIDATION,
+)
 from ludwig.data.split import get_splitter
 from ludwig.globals import HYPEROPT_STATISTICS_FILE_NAME
 from ludwig.utils.data_utils import read_csv
@@ -73,8 +84,8 @@ class Experiment:
     def __init__(self, csv_filename, tmpdir):
         self.tmpdir = tmpdir
         self.csv_file = os.path.join(tmpdir, csv_filename)
-        self.input_features = [category_feature(vocab_size=10)]
-        self.output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+        self.input_features = [category_feature(encoder={"vocab_size": 10})]
+        self.output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
         data_csv = generate_data(self.input_features, self.output_features, self.csv_file)
         self.model = self._create_model()
         test_df, train_df, val_df = obtain_df_splits(data_csv)
@@ -492,21 +503,21 @@ def test_confidence_thresholding_2thresholds_2d_vis_api(csv_filename):
     :return: None
     """
     input_features = [
-        text_feature(vocab_size=10, min_len=1, encoder="stacked_cnn"),
+        text_feature(encoder={"vocab_size": 10, "min_len": 1, "type": "stacked_cnn"}),
         number_feature(),
-        category_feature(vocab_size=10, embedding_size=5),
+        category_feature(encoder={"vocab_size": 10, "embedding_size": 5}),
         set_feature(),
-        sequence_feature(vocab_size=10, max_len=10, encoder="embed"),
+        sequence_feature(encoder={"vocab_size": 10, "max_len": 10, "type": "embed"}),
     ]
     output_features = [
-        category_feature(vocab_size=2, reduce_input="sum"),
-        category_feature(vocab_size=2, reduce_input="sum"),
+        category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
+        category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
     ]
     encoder = "parallel_cnn"
     with TemporaryDirectory() as tmpvizdir:
         # Generate test data
         data_csv = generate_data(input_features, output_features, os.path.join(tmpvizdir, csv_filename))
-        input_features[0]["encoder"] = encoder
+        input_features[0][ENCODER][TYPE] = encoder
         model = run_api_experiment(input_features, output_features)
         test_df, train_df, val_df = obtain_df_splits(data_csv)
         _, _, output_dir = model.train(
@@ -564,21 +575,21 @@ def test_confidence_thresholding_2thresholds_3d_vis_api(csv_filename):
     :return: None
     """
     input_features = [
-        text_feature(vocab_size=10, min_len=1, encoder="stacked_cnn"),
+        text_feature(encoder={"vocab_size": 10, "min_len": 1, "type": "stacked_cnn"}),
         number_feature(),
-        category_feature(vocab_size=10, embedding_size=5),
+        category_feature(encoder={"vocab_size": 10, "embedding_size": 5}),
         set_feature(),
-        sequence_feature(vocab_size=10, max_len=10, encoder="embed"),
+        sequence_feature(encoder={"vocab_size": 10, "max_len": 10, "type": "embed"}),
     ]
     output_features = [
-        category_feature(vocab_size=2, reduce_input="sum"),
-        category_feature(vocab_size=2, reduce_input="sum"),
+        category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
+        category_feature(decoder={"vocab_size": 2}, reduce_input="sum"),
     ]
     encoder = "parallel_cnn"
     with TemporaryDirectory() as tmpvizdir:
         # Generate test data
         data_csv = generate_data(input_features, output_features, os.path.join(tmpvizdir, csv_filename))
-        input_features[0]["encoder"] = encoder
+        input_features[0][ENCODER][TYPE] = encoder
         model = run_api_experiment(input_features, output_features)
         test_df, train_df, val_df = obtain_df_splits(data_csv)
         _, _, output_dir = model.train(
diff --git a/tests/integration_tests/test_whylogs.py b/tests/integration_tests/test_whylogs.py
index 04e0126598e..4a80e07b5ba 100644
--- a/tests/integration_tests/test_whylogs.py
+++ b/tests/integration_tests/test_whylogs.py
@@ -33,8 +33,8 @@ def test_whylogs_callback_local(tmpdir):
     batch_size = 8
     num_examples = 32
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     config = {
         "input_features": input_features,
@@ -67,8 +67,8 @@ def test_whylogs_callback_local(tmpdir):
 def test_whylogs_callback_dask(tmpdir):
     num_examples = 100
 
-    input_features = [sequence_feature(reduce_output="sum")]
-    output_features = [category_feature(vocab_size=2, reduce_input="sum")]
+    input_features = [sequence_feature(encoder={"reduce_output": "sum"})]
+    output_features = [category_feature(decoder={"vocab_size": 2}, reduce_input="sum")]
 
     data_csv = generate_data(
         input_features, output_features, os.path.join(tmpdir, "train.csv"), num_examples=num_examples
diff --git a/tests/integration_tests/utils.py b/tests/integration_tests/utils.py
index 38b9998dd59..222345bfdd1 100644
--- a/tests/integration_tests/utils.py
+++ b/tests/integration_tests/utils.py
@@ -34,7 +34,7 @@
 
 from ludwig.api import LudwigModel
 from ludwig.backend import LocalBackend
-from ludwig.constants import COLUMN, NAME, PROC_COLUMN, TRAINER, VECTOR
+from ludwig.constants import COLUMN, ENCODER, NAME, PROC_COLUMN, TRAINER, VECTOR
 from ludwig.data.dataset_synthesizer import build_synthetic_dataset, DATETIME_FORMATS
 from ludwig.experiment import experiment_cli
 from ludwig.features.feature_utils import compute_feature_hash
@@ -46,7 +46,6 @@
 except ImportError:
     ray = None
 
-
 logger = logging.getLogger(__name__)
 
 # Used in sequence-related unit tests (encoders, features) as well as end-to-end integration tests.
@@ -75,7 +74,6 @@
     "mt5",
 ]
 
-
 RAY_BACKEND_CONFIG = {
     "type": "ray",
     "processor": {
@@ -198,6 +196,7 @@ def generate_data(
     :param input_features: schema
     :param output_features: schema
     :param filename: path to the file where data is stored
+    :param nan_percent: percent of values in a feature to be NaN
     :return:
     """
     features = input_features + output_features
@@ -212,21 +211,38 @@ def generate_data(
     return filename
 
 
+def recursive_update(dictionary, values):
+    for k, v in values.items():
+        if isinstance(v, dict):
+            dictionary[k] = recursive_update(dictionary.get(k, {}), v)
+        else:
+            dictionary[k] = v
+    return dictionary
+
+
 def random_string(length=5):
     return uuid.uuid4().hex[:length].upper()
 
 
 def number_feature(normalization=None, **kwargs):
-    feature = {"name": "num_" + random_string(), "type": "number", "preprocessing": {"normalization": normalization}}
-    feature.update(kwargs)
+    feature = {
+        "name": "num_" + random_string(),
+        "type": "number",
+        "preprocessing": {"normalization": normalization},
+    }
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
 
 
 def category_feature(**kwargs):
-    feature = {"type": "category", "name": "category_" + random_string(), "vocab_size": 10, "embedding_size": 5}
-    feature.update(kwargs)
+    feature = {
+        "type": "category",
+        "name": "category_" + random_string(),
+        ENCODER: {"type": "dense", "vocab_size": 10, "embedding_size": 5},
+    }
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
@@ -236,21 +252,28 @@ def text_feature(**kwargs):
     feature = {
         "name": "text_" + random_string(),
         "type": "text",
-        "vocab_size": 5,
-        "min_len": 7,
-        "max_len": 7,
-        "embedding_size": 8,
-        "state_size": 8,
+        ENCODER: {
+            "type": "parallel_cnn",
+            "vocab_size": 5,
+            "min_len": 7,
+            "max_len": 7,
+            "embedding_size": 8,
+            "state_size": 8,
+        },
     }
-    feature.update(kwargs)
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
 
 
 def set_feature(**kwargs):
-    feature = {"type": "set", "name": "set_" + random_string(), "vocab_size": 10, "max_len": 5, "embedding_size": 5}
-    feature.update(kwargs)
+    feature = {
+        "type": "set",
+        "name": "set_" + random_string(),
+        ENCODER: {"type": "embed", "vocab_size": 10, "max_len": 5, "embedding_size": 5},
+    }
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
@@ -260,16 +283,18 @@ def sequence_feature(**kwargs):
     feature = {
         "type": "sequence",
         "name": "sequence_" + random_string(),
-        "vocab_size": 10,
-        "max_len": 7,
-        "encoder": "embed",
-        "embedding_size": 8,
-        "output_size": 8,
-        "state_size": 8,
-        "num_filters": 8,
-        "hidden_size": 8,
+        ENCODER: {
+            "type": "embed",
+            "vocab_size": 10,
+            "max_len": 7,
+            "embedding_size": 8,
+            "output_size": 8,
+            "state_size": 8,
+            "num_filters": 8,
+            "hidden_size": 8,
+        },
     }
-    feature.update(kwargs)
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
@@ -279,14 +304,16 @@ def image_feature(folder, **kwargs):
     feature = {
         "type": "image",
         "name": "image_" + random_string(),
-        "encoder": "resnet",
         "preprocessing": {"in_memory": True, "height": 12, "width": 12, "num_channels": 3},
-        "resnet_size": 8,
+        ENCODER: {
+            "type": "resnet",
+            "resnet_size": 8,
+            "num_filters": 8,
+            "output_size": 8,
+        },
         "destination_folder": folder,
-        "output_size": 8,
-        "num_filters": 8,
     }
-    feature.update(kwargs)
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
@@ -303,40 +330,53 @@ def audio_feature(folder, **kwargs):
             "num_filter_bands": 80,
             "audio_file_length_limit_in_s": 3.0,
         },
-        "encoder": "stacked_cnn",
-        "should_embed": False,
-        "conv_layers": [
-            {"filter_size": 400, "pool_size": 16, "num_filters": 32},
-            {"filter_size": 40, "pool_size": 10, "num_filters": 64},
-        ],
-        "output_size": 16,
+        ENCODER: {
+            "type": "stacked_cnn",
+            "should_embed": False,
+            "conv_layers": [
+                {"filter_size": 400, "pool_size": 16, "num_filters": 32},
+                {"filter_size": 40, "pool_size": 10, "num_filters": 64},
+            ],
+            "output_size": 16,
+        },
         "destination_folder": folder,
     }
-    feature.update(kwargs)
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
 
 
 def timeseries_feature(**kwargs):
-    feature = {"name": "timeseries_" + random_string(), "type": "timeseries", "max_len": 7}
-    feature.update(kwargs)
+    feature = {
+        "name": "timeseries_" + random_string(),
+        "type": "timeseries",
+        ENCODER: {"type": "parallel_cnn", "max_len": 7},
+    }
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
 
 
 def binary_feature(**kwargs):
-    feature = {"name": "binary_" + random_string(), "type": "binary"}
-    feature.update(kwargs)
+    feature = {
+        "name": "binary_" + random_string(),
+        "type": "binary",
+    }
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
 
 
 def bag_feature(**kwargs):
-    feature = {"name": "bag_" + random_string(), "type": "bag", "max_len": 5, "vocab_size": 10, "embedding_size": 5}
-    feature.update(kwargs)
+    feature = {
+        "name": "bag_" + random_string(),
+        "type": "bag",
+        ENCODER: {"type": "embed", "max_len": 5, "vocab_size": 10, "embedding_size": 5},
+    }
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
@@ -348,7 +388,7 @@ def date_feature(**kwargs):
         "type": "date",
         "preprocessing": {"datetime_format": random.choice(list(DATETIME_FORMATS.keys()))},
     }
-    feature.update(kwargs)
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
@@ -356,15 +396,21 @@ def date_feature(**kwargs):
 
 def h3_feature(**kwargs):
     feature = {"name": "h3_" + random_string(), "type": "h3"}
-    feature.update(kwargs)
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
 
 
 def vector_feature(**kwargs):
-    feature = {"type": VECTOR, "vector_size": 5, "name": "vector_" + random_string()}
-    feature.update(kwargs)
+    feature = {
+        "type": VECTOR,
+        "name": "vector_" + random_string(),
+        "preprocessing": {
+            "vector_size": 5,
+        },
+    }
+    recursive_update(feature, kwargs)
     feature[COLUMN] = feature[NAME]
     feature[PROC_COLUMN] = compute_feature_hash(feature)
     return feature
@@ -427,8 +473,8 @@ def generate_output_features_with_dependencies(main_feature, dependencies):
     """
 
     output_features = [
-        category_feature(vocab_size=2, reduce_input="sum"),
-        sequence_feature(vocab_size=10, max_len=5),
+        category_feature(decoder={"type": "classifier", "vocab_size": 2}, reduce_input="sum"),
+        sequence_feature(decoder={"type": "generator", "vocab_size": 10, "max_len": 5}),
         number_feature(),
     ]
 
@@ -453,12 +499,14 @@ def generate_output_features_with_dependencies(main_feature, dependencies):
 def generate_output_features_with_dependencies_complex():
     """Generates multiple output features specifications with dependencies."""
 
-    tf = text_feature(vocab_size=4, max_len=5, decoder="generator")
-    sf = sequence_feature(vocab_size=4, max_len=5, decoder="generator", dependencies=[tf["name"]])
+    tf = text_feature(decoder={"vocab_size": 4, "max_len": 5, "type": "generator"})
+    sf = sequence_feature(decoder={"vocab_size": 4, "max_len": 5, "type": "generator"}, dependencies=[tf["name"]])
     nf = number_feature(dependencies=[tf["name"]])
     vf = vector_feature(dependencies=[sf["name"], nf["name"]])
-    set_f = set_feature(vocab_size=4, dependencies=[tf["name"], vf["name"]])
-    cf = category_feature(vocab_size=4, dependencies=[sf["name"], nf["name"], set_f["name"]])
+    set_f = set_feature(decoder={"type": "classifier", "vocab_size": 4}, dependencies=[tf["name"], vf["name"]])
+    cf = category_feature(
+        decoder={"type": "classifier", "vocab_size": 4}, dependencies=[sf["name"], nf["name"], set_f["name"]]
+    )
 
     # The correct order ids[tf, sf, nf, vf, set_f, cf]
     # # shuffling it to test the robustness of the topological sort
diff --git a/tests/ludwig/combiners/test_combiners.py b/tests/ludwig/combiners/test_combiners.py
index 2a38b77b671..6445e3ced28 100644
--- a/tests/ludwig/combiners/test_combiners.py
+++ b/tests/ludwig/combiners/test_combiners.py
@@ -16,15 +16,13 @@
     TransformerCombiner,
 )
 from ludwig.encoders.registry import sequence_encoder_registry
-from ludwig.schema.combiners import (
-    ComparatorCombinerConfig,
-    ConcatCombinerConfig,
-    SequenceCombinerConfig,
-    SequenceConcatCombinerConfig,
-    TabNetCombinerConfig,
-    TabTransformerCombinerConfig,
-    TransformerCombinerConfig,
-)
+from ludwig.schema.combiners.comparator import ComparatorCombinerConfig
+from ludwig.schema.combiners.concat import ConcatCombinerConfig
+from ludwig.schema.combiners.sequence import SequenceCombinerConfig
+from ludwig.schema.combiners.sequence_concat import SequenceConcatCombinerConfig
+from ludwig.schema.combiners.tab_transformer import TabTransformerCombinerConfig
+from ludwig.schema.combiners.tabnet import TabNetCombinerConfig
+from ludwig.schema.combiners.transformer import TransformerCombinerConfig
 from ludwig.schema.utils import load_config
 from ludwig.utils.torch_utils import get_torch_device
 
diff --git a/tests/ludwig/encoders/test_text_encoders.py b/tests/ludwig/encoders/test_text_encoders.py
index 529fab3b714..68d27d4e202 100644
--- a/tests/ludwig/encoders/test_text_encoders.py
+++ b/tests/ludwig/encoders/test_text_encoders.py
@@ -2,8 +2,10 @@
 import torch
 
 from ludwig.encoders import text_encoders
+from tests.integration_tests.utils import slow
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -19,6 +21,7 @@ def test_albert_encoder(use_pretrained: bool, reduce_output: str, max_sequence_l
     assert outputs["encoder_output"].shape[1:] == albert_encoder.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "cls_pooled", "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -33,6 +36,7 @@ def test_bert_encoder(use_pretrained: bool, reduce_output: str, max_sequence_len
     assert outputs["encoder_output"].shape[1:] == bert.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", ["last", "sum", "mean"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -47,7 +51,7 @@ def test_xlm_encoder(use_pretrained: bool, reduce_output: str, max_sequence_leng
     assert outputs["encoder_output"].shape[1:] == xlm_encoder.output_shape
 
 
-@pytest.mark.skip(reason="Causes timeout in GitHub actions.")
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -62,6 +66,7 @@ def test_gpt_encoder(use_pretrained: bool, reduce_output: str, max_sequence_leng
     assert outputs["encoder_output"].shape[1:] == gpt_encoder.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", ["cls_pooled", "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -76,6 +81,7 @@ def test_roberta_encoder(use_pretrained: bool, reduce_output: str, max_sequence_
     assert outputs["encoder_output"].shape[1:] == roberta_encoder.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [True, False])
 @pytest.mark.parametrize("reduce_output", [None, "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -104,6 +110,7 @@ def test_distil_bert(use_pretrained: bool, reduce_output: str, max_sequence_leng
     assert outputs["encoder_output"].shape[1:] == distil_bert_encoder.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -118,6 +125,7 @@ def test_transfoxl_encoder(use_pretrained: bool, reduce_output: str, max_sequenc
     assert outputs["encoder_output"].shape[1:] == transfo.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -132,6 +140,7 @@ def test_ctrl_encoder(use_pretrained: bool, reduce_output: str, max_sequence_len
     assert outputs["encoder_output"].shape[1:] == encoder.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "cls_pooled"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -146,6 +155,7 @@ def test_camembert_encoder(use_pretrained: bool, reduce_output: str, max_sequenc
     assert outputs["encoder_output"].shape[1:] == encoder.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -160,6 +170,7 @@ def test_mt5_encoder(use_pretrained: bool, reduce_output: str, max_sequence_leng
     assert outputs["encoder_output"].shape[1:] == mt5_encoder.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -186,6 +197,7 @@ def test_longformer_encoder(use_pretrained: bool, reduce_output: str, max_sequen
     assert outputs["encoder_output"].shape[1:] == encoder.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
@@ -212,6 +224,7 @@ def test_auto_transformer_encoder(pretrained_model_name_or_path: str, reduce_out
     assert outputs["encoder_output"].shape[1:] == encoder.output_shape
 
 
+@slow
 @pytest.mark.parametrize("use_pretrained", [False])
 @pytest.mark.parametrize("reduce_output", [None, "sum"])
 @pytest.mark.parametrize("max_sequence_length", [20])
diff --git a/tests/ludwig/features/test_audio_feature.py b/tests/ludwig/features/test_audio_feature.py
index dd9fc976b48..cfdc473b354 100644
--- a/tests/ludwig/features/test_audio_feature.py
+++ b/tests/ludwig/features/test_audio_feature.py
@@ -33,13 +33,17 @@ def test_audio_input_feature(encoder: str) -> None:
             "num_filter_bands": 80,
             "audio_file_length_limit_in_s": 3.0,
         },
-        "should_embed": False,
-        "vocab": VOCAB,
-        "max_sequence_length": SEQ_SIZE,
-        "embedding_size": AUDIO_W_SIZE,
-        "encoder": encoder,
+        "encoder": {
+            "type": encoder,
+            "should_embed": False,
+            "vocab": VOCAB,
+            "max_sequence_length": SEQ_SIZE,
+            "embedding_size": AUDIO_W_SIZE,
+        },
     }
-    audio_input_feature = AudioInputFeature(audio_config).to(DEVICE)
+
+    audio_input_feature = AudioInputFeature(audio_config)
+
     audio_tensor = torch.randn([BATCH_SIZE, SEQ_SIZE, AUDIO_W_SIZE], dtype=torch.float32).to(DEVICE)
     encoder_output = audio_input_feature(audio_tensor)
     assert encoder_output["encoder_output"].shape[1:] == audio_input_feature.output_shape
diff --git a/tests/ludwig/features/test_bag_feature.py b/tests/ludwig/features/test_bag_feature.py
index 50260e3d7ad..c10b456ef5a 100644
--- a/tests/ludwig/features/test_bag_feature.py
+++ b/tests/ludwig/features/test_bag_feature.py
@@ -5,6 +5,7 @@
 import pytest
 import torch
 
+from ludwig.constants import ENCODER
 from ludwig.features.bag_feature import BagInputFeature
 from ludwig.utils.torch_utils import get_torch_device
 
@@ -23,16 +24,18 @@ def bag_config():
     return {
         "name": "bag_feature",
         "type": "bag",
-        "max_len": 5,
-        "vocab_size": 10,
-        "embedding_size": EMBEDDING_SIZE,
-        "vocab": VOCAB,
+        "encoder": {
+            "max_len": 5,
+            "vocab_size": 10,
+            "embedding_size": EMBEDDING_SIZE,
+            "vocab": VOCAB,
+        },
     }
 
 
 @pytest.mark.parametrize("encoder", ["embed"])
 def test_bag_input_feature(bag_config: Dict, encoder: str) -> None:
-    bag_config.update({"encoder": encoder})
+    bag_config[ENCODER].update({"type": encoder})
     bag_input_feature = BagInputFeature(bag_config).to(DEVICE)
     bag_tensor = torch.randn([BATCH_SIZE, SEQ_SIZE, BAG_W_SIZE], dtype=torch.float32).to(DEVICE)
     encoder_output = bag_input_feature(bag_tensor)
diff --git a/tests/ludwig/features/test_binary_feature.py b/tests/ludwig/features/test_binary_feature.py
index b03148745ae..2c311823bdd 100644
--- a/tests/ludwig/features/test_binary_feature.py
+++ b/tests/ludwig/features/test_binary_feature.py
@@ -3,6 +3,7 @@
 import pytest
 import torch
 
+from ludwig.constants import ENCODER
 from ludwig.features.binary_feature import BinaryInputFeature, BinaryOutputFeature
 from ludwig.utils.torch_utils import get_torch_device
 
@@ -21,8 +22,8 @@ def binary_config():
 
 @pytest.mark.parametrize("encoder", ["passthrough"])
 def test_binary_input_feature(binary_config: Dict, encoder: str):
-    binary_config.update({"encoder": encoder})
-    binary_input_feature = BinaryInputFeature(binary_config)
+    binary_config.update({ENCODER: {"type": encoder}})
+    binary_input_feature = BinaryInputFeature(binary_config).to(DEVICE)
     binary_tensor = torch.randn([BATCH_SIZE, BINARY_W_SIZE], dtype=torch.float32).to(DEVICE)
 
     encoder_output = binary_input_feature(binary_tensor)
@@ -31,20 +32,22 @@ def test_binary_input_feature(binary_config: Dict, encoder: str):
 
 
 def test_binary_output_feature():
-    binary_output_feature = BinaryOutputFeature(
-        {
-            "name": "binary_feature",
-            "type": "binary",
+    binary_output_config = {
+        "name": "binary_feature",
+        "type": "binary",
+        "decoder": {
+            "type": "regressor",
             "input_size": 1,
-            "loss": {
-                "positive_class_weight": 1,
-                "robust_lambda": 0,
-                "confidence_penalty": 0,
-            },
         },
-        {},
-    ).to(DEVICE)
-    combiner_outputs = {}
+        "loss": {
+            "type": "binary_weighted_cross_entropy",
+            "positive_class_weight": 1,
+            "robust_lambda": 0,
+            "confidence_penalty": 0,
+        },
+    }
+    binary_output_feature = BinaryOutputFeature(binary_output_config, {}).to(DEVICE)
+    combiner_outputs = dict()
     combiner_outputs["combiner_output"] = torch.randn([BATCH_SIZE, BINARY_W_SIZE], dtype=torch.float32).to(DEVICE)
 
     binary_output = binary_output_feature(combiner_outputs, {})
@@ -55,19 +58,21 @@ def test_binary_output_feature():
 
 
 def test_binary_output_feature_without_positive_class_weight():
-    binary_output_feature = BinaryOutputFeature(
-        {
-            "name": "binary_feature",
-            "type": "binary",
+    binary_output_config = {
+        "name": "binary_feature",
+        "type": "binary",
+        "decoder": {
+            "type": "regressor",
             "input_size": 1,
-            "loss": {
-                "positive_class_weight": None,
-                "robust_lambda": 0,
-                "confidence_penalty": 0,
-            },
         },
-        {},
-    ).to(DEVICE)
+        "loss": {
+            "type": "binary_weighted_cross_entropy",
+            "positive_class_weight": None,
+            "robust_lambda": 0,
+            "confidence_penalty": 0,
+        },
+    }
+    binary_output_feature = BinaryOutputFeature(binary_output_config, {}).to(DEVICE)
     combiner_outputs = {}
     combiner_outputs["combiner_output"] = torch.randn([BATCH_SIZE, BINARY_W_SIZE], dtype=torch.float32).to(DEVICE)
 
diff --git a/tests/ludwig/features/test_category_feature.py b/tests/ludwig/features/test_category_feature.py
index 2acce3b2bef..c9c2115ebd6 100644
--- a/tests/ludwig/features/test_category_feature.py
+++ b/tests/ludwig/features/test_category_feature.py
@@ -4,8 +4,8 @@
 import pytest
 import torch
 
+from ludwig.constants import ENCODER, TYPE
 from ludwig.features.category_feature import CategoryInputFeature
-from ludwig.models.ecd import ECD
 from ludwig.utils.torch_utils import get_torch_device
 
 BATCH_SIZE = 2
@@ -18,13 +18,15 @@ def category_config():
         "name": "category_column_name",
         "type": "category",
         "tied": None,
-        "embedding_size": 256,
-        "embeddings_on_cpu": False,
-        "pretrained_embeddings": None,
-        "embeddings_trainable": True,
-        "dropout": 0.0,
-        "vocab": ["a", "b", "c"],
-        "embedding_initializer": None,
+        "encoder": {
+            "embedding_size": 256,
+            "embeddings_on_cpu": False,
+            "pretrained_embeddings": None,
+            "embeddings_trainable": True,
+            "dropout": 0.0,
+            "vocab": ["a", "b", "c"],
+            "embedding_initializer": None,
+        },
     }
 
 
@@ -35,13 +37,13 @@ def test_category_input_feature(
 ) -> None:
     # setup image input feature definition
     category_def = deepcopy(category_config)
-    category_def["encoder"] = encoder
+    category_def[ENCODER][TYPE] = encoder
 
     # pickup any other missing parameters
     CategoryInputFeature.populate_defaults(category_def)
 
     # ensure no exceptions raised during build
-    input_feature_obj = ECD.build_single_input(category_def, None)
+    input_feature_obj = CategoryInputFeature(category_def).to(DEVICE)
 
     # check one forward pass through input feature
     input_tensor = torch.randint(0, 3, size=(BATCH_SIZE,), dtype=torch.int32).to(DEVICE)
diff --git a/tests/ludwig/features/test_image_feature.py b/tests/ludwig/features/test_image_feature.py
index f4b14c83ca3..558ce647222 100644
--- a/tests/ludwig/features/test_image_feature.py
+++ b/tests/ludwig/features/test_image_feature.py
@@ -4,11 +4,12 @@
 import pytest
 import torch
 
-from ludwig.constants import CROP_OR_PAD, INTERPOLATE
+from ludwig.constants import CROP_OR_PAD, ENCODER, INTERPOLATE, TYPE
 from ludwig.features.image_feature import _ImagePreprocessing, ImageInputFeature
-from ludwig.models.ecd import ECD
+from ludwig.utils.torch_utils import get_torch_device
 
 BATCH_SIZE = 2
+DEVICE = get_torch_device()
 
 
 @pytest.fixture(scope="module")
@@ -16,37 +17,43 @@ def image_config():
     return {
         "name": "image_column_name",
         "type": "image",
-        "encoder": "stacked_cnn",
         "tied": None,
-        "conv_layers": None,
-        "num_conv_layers": None,
-        "filter_size": 3,
-        "num_filters": 256,
-        "strides": (1, 1),
-        "padding": "valid",
-        "dilation_rate": (1, 1),
-        "conv_use_bias": True,
-        "conv_weights_initializer": "xavier_uniform",
-        "conv_bias_initializer": "zeros",
-        "conv_norm": None,
-        "conv_norm_params": None,
-        "conv_activation": "relu",
-        "conv_dropout": 0,
-        "pool_function": "max",
-        "pool_size": (2, 2),
-        "pool_strides": None,
-        "fc_layers": None,
-        "num_fc_layers": 1,
-        "output_size": 16,
-        "fc_use_bias": True,
-        "fc_weights_initializer": "xavier_uniform",
-        "fc_bias_initializer": "zeros",
-        "fc_norm": None,
-        "fc_norm_params": None,
-        "fc_activation": "relu",
-        "fc_dropout": 0,
-        "scaling": "pixel_normalization",
-        "preprocessing": {"height": 28, "width": 28, "num_channels": 1},  # example pre-processing
+        "encoder": {
+            "type": "stacked_cnn",
+            "conv_layers": None,
+            "num_conv_layers": None,
+            "filter_size": 3,
+            "num_filters": 256,
+            "strides": (1, 1),
+            "padding": "valid",
+            "dilation_rate": (1, 1),
+            "conv_use_bias": True,
+            "conv_weights_initializer": "xavier_uniform",
+            "conv_bias_initializer": "zeros",
+            "conv_norm": None,
+            "conv_norm_params": None,
+            "conv_activation": "relu",
+            "conv_dropout": 0,
+            "pool_function": "max",
+            "pool_size": (2, 2),
+            "pool_strides": None,
+            "fc_layers": None,
+            "num_fc_layers": 1,
+            "output_size": 16,
+            "fc_use_bias": True,
+            "fc_weights_initializer": "xavier_uniform",
+            "fc_bias_initializer": "zeros",
+            "fc_norm": None,
+            "fc_norm_params": None,
+            "fc_activation": "relu",
+            "fc_dropout": 0,
+        },
+        "preprocessing": {
+            "height": 28,
+            "width": 28,
+            "num_channels": 1,
+            "scaling": "pixel_normalization",
+        },
     }
 
 
@@ -60,22 +67,22 @@ def image_config():
         ("vit", 224, 224, 3),
     ],
 )
-def test_image_input_feature(image_config: Dict, encoder: str, height: int, width: int, num_channels) -> None:
+def test_image_input_feature(image_config: Dict, encoder: str, height: int, width: int, num_channels: int) -> None:
     # setup image input feature definition
     image_def = deepcopy(image_config)
-    image_def["encoder"] = encoder
-    image_def["height"] = height
-    image_def["width"] = width
-    image_def["num_channels"] = num_channels
+    image_def[ENCODER][TYPE] = encoder
+    image_def[ENCODER]["height"] = height
+    image_def[ENCODER]["width"] = width
+    image_def[ENCODER]["num_channels"] = num_channels
 
     # pickup any other missing parameters
     ImageInputFeature.populate_defaults(image_def)
 
     # ensure no exceptions raised during build
-    input_feature_obj = ECD.build_single_input(image_def, None)
+    input_feature_obj = ImageInputFeature(image_def).to(DEVICE)
 
     # check one forward pass through input feature
-    input_tensor = torch.randint(0, 256, size=(BATCH_SIZE, num_channels, height, width), dtype=torch.uint8)
+    input_tensor = torch.randint(0, 256, size=(BATCH_SIZE, num_channels, height, width), dtype=torch.uint8).to(DEVICE)
 
     encoder_output = input_feature_obj(input_tensor)
     assert encoder_output["encoder_output"].shape == (BATCH_SIZE, *input_feature_obj.output_shape)
diff --git a/tests/ludwig/features/test_number_feature.py b/tests/ludwig/features/test_number_feature.py
index 3ce8fd10425..e7c67821534 100644
--- a/tests/ludwig/features/test_number_feature.py
+++ b/tests/ludwig/features/test_number_feature.py
@@ -5,7 +5,6 @@
 import torch
 
 from ludwig.features.number_feature import NumberInputFeature
-from ludwig.models.ecd import ECD
 from ludwig.utils.torch_utils import get_torch_device
 
 BATCH_SIZE = 2
@@ -27,7 +26,7 @@ def test_number_input_feature(
     NumberInputFeature.populate_defaults(number_def)
 
     # ensure no exceptions raised during build
-    input_feature_obj = ECD.build_single_input(number_def, None).to(DEVICE)
+    input_feature_obj = NumberInputFeature(number_def).to(DEVICE)
 
     # check one forward pass through input feature
     input_tensor = torch.rand(2, dtype=torch.float32).to(DEVICE)
diff --git a/tests/ludwig/features/test_sequence_features.py b/tests/ludwig/features/test_sequence_features.py
index 3cc550eaf0b..4e2b74432e2 100644
--- a/tests/ludwig/features/test_sequence_features.py
+++ b/tests/ludwig/features/test_sequence_features.py
@@ -1,13 +1,13 @@
-from typing import List, Tuple, Union
+from typing import List, Tuple
 
 import numpy as np
 import pytest
 import torch
 import torchtext
 
-from ludwig.constants import LAST_HIDDEN, LOGITS
-from ludwig.features.sequence_feature import _SequencePreprocessing, SequenceInputFeature, SequenceOutputFeature
-from ludwig.features.text_feature import TextInputFeature, TextOutputFeature
+from ludwig.constants import LAST_HIDDEN, LOGITS, SEQUENCE, TEXT, TYPE
+from ludwig.features.sequence_feature import _SequencePreprocessing
+from ludwig.models.base import BaseModel
 from ludwig.utils.torch_utils import get_torch_device
 from tests.integration_tests.utils import ENCODERS, sequence_feature
 
@@ -42,10 +42,8 @@ def input_sequence() -> Tuple[torch.Tensor, List]:
 
 
 @pytest.mark.parametrize("encoder", ENCODERS)
-@pytest.mark.parametrize("sequence_type", [SequenceInputFeature, TextInputFeature])
-def test_sequence_input_feature(
-    input_sequence: tuple, encoder: str, sequence_type: Union[SequenceInputFeature, TextInputFeature]
-) -> None:
+@pytest.mark.parametrize("sequence_type", [SEQUENCE, TEXT])
+def test_sequence_input_feature(input_sequence: tuple, encoder: str, sequence_type: str):
     # test assumes "sequence data" has been tokenized and converted to
     # numeric representation.  Focus of this test is primarily on
     # integration with encoder with correctly sized encoder tensor and
@@ -57,16 +55,19 @@ def test_sequence_input_feature(
     # use sequence_feature() to generate baseline
     # sequence definition and then augment with
     # pre-processing metadata parameters
-    input_feature_defn = sequence_feature(
-        encoder=encoder,
-        max_len=SEQ_SIZE,
-        # augment with emulated pre-processing metadata
-        max_sequence_length=SEQ_SIZE,
-        vocab=idx2str,
+    input_feature_def = sequence_feature(
+        encoder={
+            "type": encoder,
+            "max_len": SEQ_SIZE,
+            # augment with emulated pre-processing metadata
+            "max_sequence_length": SEQ_SIZE,
+            "vocab": idx2str,
+        }
     )
+    input_feature_def[TYPE] = sequence_type
 
     # create sequence input feature object
-    input_feature_obj = sequence_type(input_feature_defn).to(DEVICE)
+    input_feature_obj = BaseModel.build_single_input(input_feature_def, None).to(DEVICE)
 
     # confirm dtype property
     assert input_feature_obj.input_dtype == torch.int32
@@ -80,12 +81,19 @@ def test_sequence_input_feature(
     assert encoder_output["encoder_output"].shape == (BATCH_SIZE, *input_feature_obj.output_shape)
 
 
-@pytest.mark.parametrize("sequence_type", [SequenceOutputFeature, TextOutputFeature])
-def test_sequence_output_feature(sequence_type: Union[SequenceOutputFeature, TextOutputFeature]):
-    output_feature_defn = sequence_feature(
-        max_len=SEQ_SIZE, max_sequence_length=SEQ_SIZE, vocab_size=VOCAB_SIZE, input_size=VOCAB_SIZE
+@pytest.mark.parametrize("sequence_type", [SEQUENCE, TEXT])
+def test_sequence_output_feature(sequence_type: str):
+    output_feature_def = sequence_feature(
+        decoder={
+            "type": "generator",
+            "max_len": SEQ_SIZE,
+            "max_sequence_length": SEQ_SIZE,
+            "vocab_size": VOCAB_SIZE,
+            "input_size": VOCAB_SIZE,
+        }
     )
-    output_feature_obj = sequence_type(output_feature_defn, {}).to(DEVICE)
+    output_feature_def[TYPE] = sequence_type
+    output_feature_obj = BaseModel.build_single_output(output_feature_def, None).to(DEVICE)
     combiner_outputs = {}
     combiner_outputs["combiner_output"] = torch.randn([BATCH_SIZE, SEQ_SIZE, VOCAB_SIZE], dtype=torch.float32).to(
         DEVICE
diff --git a/tests/ludwig/features/test_set_feature.py b/tests/ludwig/features/test_set_feature.py
index 1f3936f5584..d4c2aedac75 100644
--- a/tests/ludwig/features/test_set_feature.py
+++ b/tests/ludwig/features/test_set_feature.py
@@ -4,8 +4,8 @@
 import pytest
 import torch
 
+from ludwig.constants import ENCODER
 from ludwig.features.set_feature import SetInputFeature
-from ludwig.models.ecd import ECD
 from ludwig.utils.torch_utils import get_torch_device
 
 BATCH_SIZE = 2
@@ -17,31 +17,30 @@ def set_config():
     return {
         "name": "set_column_name",
         "type": "set",
-        "vocab": ["a", "b", "c"],
-        "representation": "dense",
-        "encoder": "embed",
-        "embedding_size": 50,
-        "embeddings_trainable": True,
-        "pretrained_embeddings": None,
-        "embeddings_on_cpu": False,
-        "fc_layers": None,
-        "num_fc_layers": 0,
-        "output_size": 0,
-        "use_bias": True,
-        "weights_initializer": "uniform",
-        "bias_initializer": "zeros",
-        "norm": None,
-        "norm_params": None,
-        "activation": "relu",
-        "dropout": 0.0,
-        "reduce_output": "sum",
         "tied": None,
+        "encoder": {
+            "type": "embed",
+            "vocab": ["a", "b", "c"],
+            "representation": "dense",
+            "embedding_size": 50,
+            "embeddings_trainable": True,
+            "pretrained_embeddings": None,
+            "embeddings_on_cpu": False,
+            "fc_layers": None,
+            "num_fc_layers": 0,
+            "use_bias": True,
+            "weights_initializer": "uniform",
+            "bias_initializer": "zeros",
+            "norm": None,
+            "norm_params": None,
+            "activation": "relu",
+            "dropout": 0.0,
+            "reduce_output": "sum",
+        },
     }
 
 
-def test_set_input_feature(
-    set_config: Dict,
-) -> None:
+def test_set_input_feature(set_config: Dict) -> None:
     # setup image input feature definition
     set_def = deepcopy(set_config)
 
@@ -49,10 +48,10 @@ def test_set_input_feature(
     SetInputFeature.populate_defaults(set_def)
 
     # ensure no exceptions raised during build
-    input_feature_obj = ECD.build_single_input(set_def, None).to(DEVICE)
+    input_feature_obj = SetInputFeature(set_def).to(DEVICE)
 
     # check one forward pass through input feature
-    input_tensor = torch.randint(0, 2, size=(BATCH_SIZE, len(set_def["vocab"])), dtype=torch.int64).to(DEVICE)
+    input_tensor = torch.randint(0, 2, size=(BATCH_SIZE, len(set_def[ENCODER]["vocab"])), dtype=torch.int64).to(DEVICE)
 
     encoder_output = input_feature_obj(input_tensor)
     assert encoder_output["encoder_output"].shape == (BATCH_SIZE, *input_feature_obj.output_shape)
diff --git a/tests/ludwig/features/test_timeseries_feature.py b/tests/ludwig/features/test_timeseries_feature.py
index 2e2e5853c45..e7392c7a640 100644
--- a/tests/ludwig/features/test_timeseries_feature.py
+++ b/tests/ludwig/features/test_timeseries_feature.py
@@ -3,6 +3,7 @@
 import pytest
 import torch
 
+from ludwig.constants import ENCODER, TYPE
 from ludwig.features.timeseries_feature import TimeseriesInputFeature
 from ludwig.utils.torch_utils import get_torch_device
 
@@ -18,19 +19,22 @@ def timeseries_config():
     return {
         "name": "timeseries_12",
         "type": "timeseries",
-        "max_len": MAX_LEN,
-        "embedding_size": EMBEDDING_SIZE,
-        "max_sequence_length": SEQ_SIZE,
-        "output_size": 8,
-        "state_size": 8,
-        "num_filters": 8,
-        "hidden_size": 8,
+        "encoder": {
+            "max_len": MAX_LEN,
+            "embedding_size": EMBEDDING_SIZE,
+            "max_sequence_length": SEQ_SIZE,
+            "output_size": 8,
+            "state_size": 8,
+            "num_filters": 8,
+            "hidden_size": 8,
+        },
     }
 
 
 @pytest.mark.parametrize("encoder", ["rnn", "stacked_cnn", "parallel_cnn"])
 def test_timeseries_input_feature(timeseries_config: Dict, encoder: str) -> None:
-    timeseries_config.update({"encoder": encoder})
+    timeseries_config[ENCODER][TYPE] = encoder
+
     timeseries_input_feature = TimeseriesInputFeature(timeseries_config).to(DEVICE)
     timeseries_tensor = torch.randn([SEQ_SIZE, TIMESERIES_W_SIZE], dtype=torch.float32).to(DEVICE)
     encoder_output = timeseries_input_feature(timeseries_tensor)
diff --git a/tests/ludwig/marshmallow/test_fields_misc.py b/tests/ludwig/marshmallow/test_fields_misc.py
index 428a23199ff..33e4bc16e8d 100644
--- a/tests/ludwig/marshmallow/test_fields_misc.py
+++ b/tests/ludwig/marshmallow/test_fields_misc.py
@@ -108,7 +108,7 @@ def test_FloatRangeTupleDataclassField():
 
     # Test dimensional mismatch:
     with pytest.raises(MarshmallowValidationError):
-        schema_utils.FloatRangeTupleDataclassField(N=3, default=(1, 1))
+        schema_utils.FloatRangeTupleDataclassField(n=3, default=(1, 1))
 
     # Test default schema creation:
     @dataclass
@@ -129,7 +129,7 @@ class CustomTestSchema(schema_utils.BaseMarshmallowConfig):
     # Test non-default schema (N=3, other custom metadata):
     @dataclass
     class CustomTestSchema(schema_utils.BaseMarshmallowConfig):
-        foo: Tuple[float, float] = schema_utils.FloatRangeTupleDataclassField(N=3, default=(1, 1, 1), min=-10, max=10)
+        foo: Tuple[float, float] = schema_utils.FloatRangeTupleDataclassField(n=3, default=(1, 1, 1), min=-10, max=10)
 
     assert CustomTestSchema.Schema().load({}).foo == (1, 1, 1)
     assert CustomTestSchema.Schema().load({"foo": [2, 2, 2]}).foo == (2, 2, 2)
diff --git a/tests/ludwig/utils/test_backward_compatibility.py b/tests/ludwig/utils/test_backward_compatibility.py
index da07963318a..34267e4240b 100644
--- a/tests/ludwig/utils/test_backward_compatibility.py
+++ b/tests/ludwig/utils/test_backward_compatibility.py
@@ -1,4 +1,9 @@
-from ludwig.utils.backward_compatibility import _upgrade_feature, _upgrade_preprocessing_split
+from ludwig.constants import INPUT_FEATURES, OUTPUT_FEATURES
+from ludwig.utils.backward_compatibility import (
+    _upgrade_encoder_decoder_params,
+    _upgrade_feature,
+    _upgrade_preprocessing_split,
+)
 
 
 def test_preprocessing_backward_compatibility():
@@ -93,3 +98,149 @@ def test_audio_feature_backward_compatibility():
             "num_filter_bands": 80,
         },
     }
+
+
+def test_encoder_decoder_backwards_compatibility():
+    old_config = {
+        "input_features": [
+            {
+                "name": "text_feature",
+                "type": "text",
+                "preprocessing": {
+                    "missing_value_strategy": "drop_row",
+                },
+                "encoder": "rnn",
+                "bidirectional": True,
+                "representation": "dense",
+                "num_layers": 2,
+            },
+            {
+                "name": "image_feature_1",
+                "type": "image",
+                "preprocessing": {
+                    "height": 7.5,
+                    "width": 7.5,
+                    "num_channels": 4,
+                },
+                "encoder": "resnet",
+                "num_channels": 4,
+                "dropout": 0.1,
+                "resnet_size": 100,
+            },
+            {
+                "name": "image_feature_2",
+                "type": "image",
+                "tied": "image_feature_1",
+                "preprocessing": {
+                    "height": 7.5,
+                    "width": 7.5,
+                    "num_channels": 4,
+                },
+                "encoder": "resnet",
+            },
+        ],
+        "output_features": [
+            {
+                "name": "category_feature",
+                "type": "category",
+                "top_k": 3,
+                "preprocessing": {
+                    "missing_value_strategy": "backfill",
+                },
+                "decoder": "classifier",
+                "num_classes": 10,
+                "use_bias": False,
+            },
+            {
+                "name": "binary_feature",
+                "type": "binary",
+                "dependencies": ["category_feature"],
+                "loss": {
+                    "type": "cross_entropy",
+                },
+                "reduce_dependencies": "mean",
+                "decoder": "regressor",
+                "use_bias": True,
+                "bias_initializer": "constant",
+            },
+        ],
+    }
+
+    for feature in old_config[INPUT_FEATURES]:
+        _upgrade_encoder_decoder_params(feature, True)
+
+    for feature in old_config[OUTPUT_FEATURES]:
+        _upgrade_encoder_decoder_params(feature, False)
+
+    assert old_config == {
+        "input_features": [
+            {
+                "name": "text_feature",
+                "type": "text",
+                "preprocessing": {
+                    "missing_value_strategy": "drop_row",
+                },
+                "encoder": {
+                    "type": "rnn",
+                    "bidirectional": True,
+                    "representation": "dense",
+                    "num_layers": 2,
+                },
+            },
+            {
+                "name": "image_feature_1",
+                "type": "image",
+                "preprocessing": {
+                    "height": 7.5,
+                    "width": 7.5,
+                    "num_channels": 4,
+                },
+                "encoder": {
+                    "type": "resnet",
+                    "num_channels": 4,
+                    "dropout": 0.1,
+                    "resnet_size": 100,
+                },
+            },
+            {
+                "name": "image_feature_2",
+                "type": "image",
+                "tied": "image_feature_1",
+                "preprocessing": {
+                    "height": 7.5,
+                    "width": 7.5,
+                    "num_channels": 4,
+                },
+                "encoder": {"type": "resnet"},
+            },
+        ],
+        "output_features": [
+            {
+                "name": "category_feature",
+                "type": "category",
+                "top_k": 3,
+                "preprocessing": {
+                    "missing_value_strategy": "backfill",
+                },
+                "decoder": {
+                    "type": "classifier",
+                    "num_classes": 10,
+                    "use_bias": False,
+                },
+            },
+            {
+                "name": "binary_feature",
+                "type": "binary",
+                "dependencies": ["category_feature"],
+                "loss": {
+                    "type": "cross_entropy",
+                },
+                "reduce_dependencies": "mean",
+                "decoder": {
+                    "type": "regressor",
+                    "use_bias": True,
+                    "bias_initializer": "constant",
+                },
+            },
+        ],
+    }
diff --git a/tests/ludwig/utils/test_defaults.py b/tests/ludwig/utils/test_defaults.py
index 7913bac10ed..904f93ec14c 100644
--- a/tests/ludwig/utils/test_defaults.py
+++ b/tests/ludwig/utils/test_defaults.py
@@ -5,8 +5,11 @@
 
 from ludwig.constants import (
     CATEGORY,
+    DECODER,
     DEFAULTS,
+    DEPENDENCIES,
     DROP_ROW,
+    ENCODER,
     EVAL_BATCH_SIZE,
     EXECUTOR,
     FILL_WITH_MODE,
@@ -19,14 +22,19 @@
     NUMBER,
     OUTPUT_FEATURES,
     PREPROCESSING,
+    REDUCE_DEPENDENCIES,
+    REDUCE_INPUT,
     SCHEDULER,
     SPLIT,
+    SUM,
+    TIED,
+    TOP_K,
     TRAINER,
     TYPE,
 )
 from ludwig.schema.trainer import ECDTrainerConfig
 from ludwig.utils.defaults import BASE_PREPROCESSING_SPLIT_CONFIG, merge_with_defaults
-from ludwig.utils.misc_utils import merge_dict
+from ludwig.utils.misc_utils import merge_dict, set_default_values
 from tests.integration_tests.utils import (
     binary_feature,
     category_feature,
@@ -268,6 +276,42 @@ def test_deprecated_split_aliases(stratify, force_split):
         assert split.get("column") == stratify
 
 
+def test_set_default_values():
+    config = {
+        INPUT_FEATURES: [number_feature(encoder={"max_sequence_length": 10})],
+        OUTPUT_FEATURES: [category_feature(decoder={})],
+    }
+
+    assert TIED not in config[INPUT_FEATURES][0]
+    assert TOP_K not in config[OUTPUT_FEATURES][0]
+    assert DEPENDENCIES not in config[OUTPUT_FEATURES][0]
+    assert REDUCE_INPUT not in config[OUTPUT_FEATURES][0]
+    assert REDUCE_DEPENDENCIES not in config[OUTPUT_FEATURES][0]
+
+    set_default_values(config[INPUT_FEATURES][0], {ENCODER: {TYPE: "passthrough"}, TIED: None})
+
+    set_default_values(
+        config[OUTPUT_FEATURES][0],
+        {
+            DECODER: {
+                TYPE: "classifier",
+            },
+            TOP_K: 3,
+            DEPENDENCIES: [],
+            REDUCE_INPUT: SUM,
+            REDUCE_DEPENDENCIES: SUM,
+        },
+    )
+
+    assert config[INPUT_FEATURES][0][ENCODER][TYPE] == "passthrough"
+    assert config[INPUT_FEATURES][0][TIED] is None
+    assert config[OUTPUT_FEATURES][0][DECODER][TYPE] == "classifier"
+    assert config[OUTPUT_FEATURES][0][TOP_K] == 3
+    assert config[OUTPUT_FEATURES][0][DEPENDENCIES] == []
+    assert config[OUTPUT_FEATURES][0][REDUCE_INPUT] == SUM
+    assert config[OUTPUT_FEATURES][0][REDUCE_DEPENDENCIES] == SUM
+
+
 def test_merge_with_defaults():
     # configuration with legacy parameters
     legacy_config_format = {
@@ -324,6 +368,7 @@ def test_merge_with_defaults():
             {
                 "type": "number",
                 "name": "number_input_feature",
+                "encoder": {"type": "passthrough"},
                 "column": "number_input_feature",
                 "proc_column": "number_input_feature_mZFLky",
                 "tied": None,
@@ -331,12 +376,14 @@ def test_merge_with_defaults():
             {
                 "type": "image",
                 "name": "image_input_feature",
-                "encoder": "stacked_cnn",
-                "conv_layers": [
-                    {"num_filters": 32, "pool_size": 2, "pool_stride": 2, "use_bias": False},
-                    {"num_filters": 64, "pool_size": 2, "pool_stride": 2},
-                ],
-                "conv_use_bias": True,
+                "encoder": {
+                    "type": "stacked_cnn",
+                    "conv_layers": [
+                        {"num_filters": 32, "pool_size": 2, "pool_stride": 2, "use_bias": False},
+                        {"num_filters": 64, "pool_size": 2, "pool_stride": 2},
+                    ],
+                    "conv_use_bias": True,
+                },
                 "column": "image_input_feature",
                 "proc_column": "image_input_feature_mZFLky",
                 "tied": None,
@@ -348,9 +395,12 @@ def test_merge_with_defaults():
                 "type": "number",
                 "name": "number_output_feature",
                 "column": "number_output_feature",
+                "decoder": {
+                    "type": "regressor",
+                    "clip": None,
+                },
                 "proc_column": "number_output_feature_mZFLky",
                 "loss": {"type": "mean_squared_error", "weight": 1},
-                "clip": None,
                 "dependencies": [],
                 "reduce_input": "sum",
                 "reduce_dependencies": "sum",
@@ -424,6 +474,7 @@ def test_merge_with_defaults():
                     "lowercase": True,
                     "missing_value_strategy": "fill_with_const",
                     "fill_value": "<UNK>",
+                    "computed_fill_value": "<UNK>",
                 }
             },
             "category": {
@@ -432,6 +483,7 @@ def test_merge_with_defaults():
                     "lowercase": False,
                     "missing_value_strategy": "fill_with_const",
                     "fill_value": "<UNK>",
+                    "computed_fill_value": "<UNK>",
                 }
             },
             "set": {
@@ -441,6 +493,7 @@ def test_merge_with_defaults():
                     "lowercase": False,
                     "missing_value_strategy": "fill_with_const",
                     "fill_value": "<UNK>",
+                    "computed_fill_value": "<UNK>",
                 }
             },
             "bag": {
@@ -450,11 +503,24 @@ def test_merge_with_defaults():
                     "lowercase": False,
                     "missing_value_strategy": "fill_with_const",
                     "fill_value": "<UNK>",
+                    "computed_fill_value": "<UNK>",
+                }
+            },
+            "binary": {
+                PREPROCESSING: {
+                    "missing_value_strategy": "fill_with_false",
+                    "computed_fill_value": None,
+                    "fallback_true_label": None,
+                    "fill_value": None,
                 }
             },
-            "binary": {PREPROCESSING: {"missing_value_strategy": "fill_with_false"}},
             "number": {
-                PREPROCESSING: {"missing_value_strategy": "fill_with_const", "fill_value": 0, "normalization": None}
+                PREPROCESSING: {
+                    "computed_fill_value": 0.0,
+                    "missing_value_strategy": "fill_with_const",
+                    "fill_value": 0.0,
+                    "normalization": None,
+                }
             },
             "sequence": {
                 PREPROCESSING: {
@@ -468,20 +534,27 @@ def test_merge_with_defaults():
                     "vocab_file": None,
                     "missing_value_strategy": "fill_with_const",
                     "fill_value": "<UNK>",
+                    "computed_fill_value": "<UNK>",
                 }
             },
             "timeseries": {
                 PREPROCESSING: {
                     "timeseries_length_limit": 256,
-                    "padding_value": 0,
+                    "padding_value": 0.0,
                     "padding": "right",
                     "tokenizer": "space",
                     "missing_value_strategy": "fill_with_const",
+                    "computed_fill_value": "",
                     "fill_value": "",
                 }
             },
             "image": {
                 PREPROCESSING: {
+                    "computed_fill_value": None,
+                    "fill_value": None,
+                    "height": None,
+                    "width": None,
+                    "num_channels": None,
                     "missing_value_strategy": "backfill",
                     "in_memory": True,
                     "resize_method": "interpolate",
@@ -499,7 +572,7 @@ def test_merge_with_defaults():
                     "audio_file_length_limit_in_s": 7.5,
                     "missing_value_strategy": "backfill",
                     "in_memory": True,
-                    "padding_value": 0,
+                    "padding_value": 0.0,
                     "norm": None,
                     "type": "fbank",
                     "window_length_in_s": 0.04,
@@ -507,13 +580,33 @@ def test_merge_with_defaults():
                     "num_fft_points": None,
                     "window_type": "hamming",
                     "num_filter_bands": 80,
+                    "computed_fill_value": None,
+                    "fill_value": None,
+                }
+            },
+            "h3": {
+                PREPROCESSING: {
+                    "missing_value_strategy": "fill_with_const",
+                    "fill_value": 576495936675512319,
+                    "computed_fill_value": 576495936675512319,
                 }
             },
-            "h3": {PREPROCESSING: {"missing_value_strategy": "fill_with_const", "fill_value": 576495936675512319}},
             "date": {
-                PREPROCESSING: {"missing_value_strategy": "fill_with_const", "fill_value": "", "datetime_format": None}
+                PREPROCESSING: {
+                    "missing_value_strategy": "fill_with_const",
+                    "computed_fill_value": "",
+                    "fill_value": "",
+                    "datetime_format": None,
+                }
+            },
+            "vector": {
+                PREPROCESSING: {
+                    "missing_value_strategy": "fill_with_const",
+                    "computed_fill_value": "",
+                    "fill_value": "",
+                    "vector_size": None,
+                }
             },
-            "vector": {PREPROCESSING: {"missing_value_strategy": "fill_with_const", "fill_value": ""}},
         },
         "combiner": {
             "type": "concat",
diff --git a/tests/ludwig/utils/test_trainer_utils.py b/tests/ludwig/utils/test_trainer_utils.py
index 179e2e2d8a3..6efe4f7209f 100644
--- a/tests/ludwig/utils/test_trainer_utils.py
+++ b/tests/ludwig/utils/test_trainer_utils.py
@@ -10,7 +10,15 @@
 def test_progress_tracker_empty():
     output_features = LudwigFeatureDict()
     output_features["category_feature"] = CategoryOutputFeature(
-        {"name": "category_feature", "input_size": 10, "num_classes": 3}, {}
+        {
+            "name": "category_feature",
+            "decoder": {
+                "type": "classifier",
+                "input_size": 10,
+                "num_classes": 3,
+            },
+        },
+        {},
     )
 
     progress_tracker = trainer_utils.get_new_progress_tracker(
@@ -38,7 +46,15 @@ def test_progress_tracker_empty():
 def test_progress_tracker():
     output_features = LudwigFeatureDict()
     output_features["category_feature"] = CategoryOutputFeature(
-        {"name": "category_feature", "input_size": 10, "num_classes": 3}, {}
+        {
+            "name": "category_feature",
+            "decoder": {
+                "type": "classifier",
+                "input_size": 10,
+                "num_classes": 3,
+            },
+        },
+        {},
     )
 
     progress_tracker = trainer_utils.get_new_progress_tracker(
diff --git a/tests/ludwig/utils/test_validate_config_combiner.py b/tests/ludwig/utils/test_validate_config_combiner.py
index 8f97b3b667d..276e4de4606 100644
--- a/tests/ludwig/utils/test_validate_config_combiner.py
+++ b/tests/ludwig/utils/test_validate_config_combiner.py
@@ -10,7 +10,7 @@
 def test_config_tabnet(eval_batch_size):
     config = {
         "input_features": [
-            category_feature(vocab_size=2, reduce_input="sum"),
+            category_feature(encoder={"type": "dense", "vocab_size": 2}, reduce_input="sum"),
             number_feature(),
         ],
         "output_features": [binary_feature(weight_regularization=None)],
@@ -48,7 +48,7 @@ def test_config_tabnet(eval_batch_size):
 def test_config_bad_combiner():
     config = {
         "input_features": [
-            category_feature(vocab_size=2, reduce_input="sum"),
+            category_feature(encoder={"type": "dense", "vocab_size": 2}, reduce_input="sum"),
             number_feature(),
         ],
         "output_features": [binary_feature(weight_regularization=None)],
@@ -96,7 +96,7 @@ def test_config_bad_combiner():
 def test_config_bad_combiner_types_enums():
     config = {
         "input_features": [
-            category_feature(vocab_size=2, reduce_input="sum"),
+            category_feature(encoder={"type": "dense", "vocab_size": 2}, reduce_input="sum"),
             number_feature(),
         ],
         "output_features": [binary_feature(weight_regularization=None)],
@@ -108,13 +108,13 @@ def test_config_bad_combiner_types_enums():
 
     # Test weights initializer:
     config["combiner"]["weights_initializer"] = {"test": "fail"}
-    with pytest.raises(ValidationError, match=r"{'test': 'fail'} is not of*"):
+    with pytest.raises(ValidationError, match=r"'type' is a required property"):
         validate_config(config)
     config["combiner"]["weights_initializer"] = "fail"
     with pytest.raises(ValidationError, match=r"'fail' is not of*"):
         validate_config(config)
     config["combiner"]["weights_initializer"] = {}
-    with pytest.raises(ValidationError, match=r"Failed validating 'type'"):
+    with pytest.raises(ValidationError, match=r"'type' is a required property"):
         validate_config(config)
     config["combiner"]["weights_initializer"] = {"type": "fail"}
     with pytest.raises(ValidationError, match=r"'fail' is not one of*"):
@@ -130,7 +130,7 @@ def test_config_bad_combiner_types_enums():
     with pytest.raises(ValidationError, match=r"'fail' is not of*"):
         validate_config(config)
     config["combiner"]["bias_initializer"] = {}
-    with pytest.raises(ValidationError, match=r"Failed validating 'type'"):
+    with pytest.raises(ValidationError, match=r"'type' is a required property"):
         validate_config(config)
     config["combiner"]["bias_initializer"] = {"type": "fail"}
     with pytest.raises(ValidationError, match=r"'fail' is not one of*"):
diff --git a/tests/ludwig/utils/test_validate_config_features.py b/tests/ludwig/utils/test_validate_config_features.py
index 493f9db2e72..e3310a45c56 100644
--- a/tests/ludwig/utils/test_validate_config_features.py
+++ b/tests/ludwig/utils/test_validate_config_features.py
@@ -8,10 +8,10 @@
 def test_config_input_output_features():
     config = {
         "input_features": [
-            category_feature(),
-            number_feature(),
+            category_feature(encoder={"type": "dense"}),
+            number_feature(encoder={"type": "passthrough"}),
         ],
-        "output_features": [binary_feature()],
+        "output_features": [binary_feature(decoder={"type": "regressor"})],
     }
 
     validate_config(config)
diff --git a/tests/ludwig/utils/test_validate_config_misc.py b/tests/ludwig/utils/test_validate_config_misc.py
index b4b2237e795..ecb6825f014 100644
--- a/tests/ludwig/utils/test_validate_config_misc.py
+++ b/tests/ludwig/utils/test_validate_config_misc.py
@@ -38,28 +38,28 @@
 
 def test_config_features():
     all_input_features = [
-        audio_feature("/tmp/destination_folder"),
-        bag_feature(),
-        binary_feature(),
-        category_feature(),
-        date_feature(),
-        h3_feature(),
-        image_feature("/tmp/destination_folder"),
-        number_feature(),
-        sequence_feature(),
-        set_feature(),
-        text_feature(),
-        timeseries_feature(),
-        vector_feature(),
+        audio_feature("/tmp/destination_folder", encoder={"type": "parallel_cnn"}),
+        bag_feature(encoder={"type": "embed"}),
+        binary_feature(encoder={"type": "passthrough"}),
+        category_feature(encoder={"type": "dense"}),
+        date_feature(encoder={"type": "embed"}),
+        h3_feature(encoder={"type": "embed"}),
+        image_feature("/tmp/destination_folder", encoder={"type": "stacked_cnn"}),
+        number_feature(encoder={"type": "passthrough"}),
+        sequence_feature(encoder={"type": "parallel_cnn"}),
+        set_feature(encoder={"type": "embed"}),
+        text_feature(encoder={"type": "parallel_cnn"}),
+        timeseries_feature(encoder={"type": "parallel_cnn"}),
+        vector_feature(encoder={"type": "dense"}),
     ]
     all_output_features = [
-        binary_feature(),
-        category_feature(),
-        number_feature(),
-        sequence_feature(),
-        set_feature(),
-        text_feature(),
-        vector_feature(),
+        binary_feature(decoder={"type": "regressor"}),
+        category_feature(decoder={"type": "classifier"}),
+        number_feature(decoder={"type": "regressor"}),
+        sequence_feature(decoder={"type": "generator"}),
+        set_feature(decoder={"type": "classifier"}),
+        text_feature(decoder={"type": "generator"}),
+        vector_feature(decoder={"type": "projector"}),
     ]
 
     # validate config with all features
@@ -92,10 +92,10 @@ def test_config_encoders():
     for encoder in ENCODERS:
         config = {
             "input_features": [
-                sequence_feature(reduce_output="sum", encoder=encoder),
+                sequence_feature(encoder={"type": encoder, "reduce_output": "sum"}),
                 image_feature("/tmp/destination_folder"),
             ],
-            "output_features": [category_feature(vocab_size=2, reduce_input="sum")],
+            "output_features": [category_feature(decoder={"type": "classifier", "vocab_size": 2}, reduce_input="sum")],
             "combiner": {"type": "concat", "output_size": 14},
         }
         validate_config(config)
@@ -104,7 +104,7 @@ def test_config_encoders():
 def test_config_tabnet():
     config = {
         "input_features": [
-            category_feature(vocab_size=2, reduce_input="sum"),
+            category_feature(encoder={"type": "dense", "vocab_size": 2}, reduce_input="sum"),
             number_feature(),
         ],
         "output_features": [binary_feature(weight_regularization=None)],
@@ -141,7 +141,7 @@ def test_config_tabnet():
 def test_config_bad_feature_type():
     config = {
         "input_features": [{"name": "foo", "type": "fake"}],
-        "output_features": [category_feature(vocab_size=2, reduce_input="sum")],
+        "output_features": [category_feature(encoder={"vocab_size": 2}, reduce_input="sum")],
         "combiner": {"type": "concat", "output_size": 14},
     }
 
@@ -151,8 +151,8 @@ def test_config_bad_feature_type():
 
 def test_config_bad_encoder_name():
     config = {
-        "input_features": [sequence_feature(reduce_output="sum", encoder="fake")],
-        "output_features": [category_feature(vocab_size=2, reduce_input="sum")],
+        "input_features": [sequence_feature(encoder={"type": "fake", "reduce_output": "sum"})],
+        "output_features": [category_feature(decoder={"type": "classifier", "vocab_size": 2}, reduce_input="sum")],
         "combiner": {"type": "concat", "output_size": 14},
     }
 
@@ -163,7 +163,7 @@ def test_config_bad_encoder_name():
 def test_config_bad_preprocessing_param():
     config = {
         "input_features": [
-            sequence_feature(reduce_output="sum", encoder="fake"),
+            sequence_feature(encoder={"type": "parallel_cnn", "reduce_output": "sum"}),
             image_feature(
                 "/tmp/destination_folder",
                 preprocessing={
@@ -175,11 +175,11 @@ def test_config_bad_preprocessing_param():
                 },
             ),
         ],
-        "output_features": [category_feature(vocab_size=2, reduce_input="sum")],
+        "output_features": [category_feature(encoder={"vocab_size": 2}, reduce_input="sum")],
         "combiner": {"type": "concat", "output_size": 14},
     }
 
-    with pytest.raises(ValidationError, match=r"^'fake' is not one of .*"):
+    with pytest.raises(ValidationError, match=r"^Additional properties are not allowed .*"):
         validate_config(config)
 
 
@@ -211,19 +211,31 @@ def test_config_fill_values():
 def test_validate_with_preprocessing_defaults():
     config = {
         "input_features": [
-            audio_feature("/tmp/destination_folder", preprocessing=AudioFeatureMixin.preprocessing_defaults()),
-            bag_feature(preprocessing=BagFeatureMixin.preprocessing_defaults()),
-            binary_feature(preprocessing=BinaryFeatureMixin.preprocessing_defaults()),
-            category_feature(preprocessing=CategoryFeatureMixin.preprocessing_defaults()),
-            date_feature(preprocessing=DateFeatureMixin.preprocessing_defaults()),
-            h3_feature(preprocessing=H3FeatureMixin.preprocessing_defaults()),
-            image_feature("/tmp/destination_folder", preprocessing=ImageFeatureMixin.preprocessing_defaults()),
-            number_feature(preprocessing=NumberFeatureMixin.preprocessing_defaults()),
-            sequence_feature(preprocessing=SequenceFeatureMixin.preprocessing_defaults()),
-            set_feature(preprocessing=SetFeatureMixin.preprocessing_defaults()),
-            text_feature(preprocessing=TextFeatureMixin.preprocessing_defaults()),
-            timeseries_feature(preprocessing=TimeseriesFeatureMixin.preprocessing_defaults()),
-            vector_feature(preprocessing=VectorFeatureMixin.preprocessing_defaults()),
+            audio_feature(
+                "/tmp/destination_folder",
+                preprocessing=AudioFeatureMixin.preprocessing_defaults(),
+                encoder={"type": "parallel_cnn"},
+            ),
+            bag_feature(preprocessing=BagFeatureMixin.preprocessing_defaults(), encoder={"type": "embed"}),
+            binary_feature(preprocessing=BinaryFeatureMixin.preprocessing_defaults(), encoder={"type": "passthrough"}),
+            category_feature(preprocessing=CategoryFeatureMixin.preprocessing_defaults(), encoder={"type": "dense"}),
+            date_feature(preprocessing=DateFeatureMixin.preprocessing_defaults(), encoder={"type": "embed"}),
+            h3_feature(preprocessing=H3FeatureMixin.preprocessing_defaults(), encoder={"type": "embed"}),
+            image_feature(
+                "/tmp/destination_folder",
+                preprocessing=ImageFeatureMixin.preprocessing_defaults(),
+                encoder={"type": "stacked_cnn"},
+            ),
+            number_feature(preprocessing=NumberFeatureMixin.preprocessing_defaults(), encoder={"type": "passthrough"}),
+            sequence_feature(
+                preprocessing=SequenceFeatureMixin.preprocessing_defaults(), encoder={"type": "parallel_cnn"}
+            ),
+            set_feature(preprocessing=SetFeatureMixin.preprocessing_defaults(), encoder={"type": "embed"}),
+            text_feature(preprocessing=TextFeatureMixin.preprocessing_defaults(), encoder={"type": "parallel_cnn"}),
+            timeseries_feature(
+                preprocessing=TimeseriesFeatureMixin.preprocessing_defaults(), encoder={"type": "parallel_cnn"}
+            ),
+            vector_feature(preprocessing=VectorFeatureMixin.preprocessing_defaults(), encoder={"type": "dense"}),
         ],
         "output_features": [{"name": "target", "type": "category"}],
         TRAINER: {
diff --git a/tests/ludwig/utils/test_validate_config_trainer.py b/tests/ludwig/utils/test_validate_config_trainer.py
index dca892a9ebf..5867b2af2e2 100644
--- a/tests/ludwig/utils/test_validate_config_trainer.py
+++ b/tests/ludwig/utils/test_validate_config_trainer.py
@@ -14,7 +14,7 @@
 def test_config_trainer_empty_null_and_default():
     config = {
         "input_features": [
-            category_feature(vocab_size=2, reduce_input="sum"),
+            category_feature(encoder={"type": "dense", "vocab_size": 2}, reduce_input="sum"),
             number_feature(),
         ],
         "output_features": [binary_feature(weight_regularization=None)],
@@ -36,7 +36,7 @@ def test_config_trainer_empty_null_and_default():
 def test_config_trainer_bad_optimizer():
     config = {
         "input_features": [
-            category_feature(vocab_size=2, reduce_input="sum"),
+            category_feature(encoder={"type": "dense", "vocab_size": 2}, reduce_input="sum"),
             number_feature(),
         ],
         "output_features": [binary_feature(weight_regularization=None)],
@@ -73,7 +73,7 @@ def test_config_trainer_bad_optimizer():
 def test_optimizer_property_validation():
     config = {
         "input_features": [
-            category_feature(vocab_size=2, reduce_input="sum"),
+            category_feature(encoder={"type": "dense", "vocab_size": 2}, reduce_input="sum"),
             number_feature(),
         ],
         "output_features": [binary_feature(weight_regularization=None)],
@@ -111,7 +111,7 @@ def test_optimizer_property_validation():
 def test_clipper_property_validation():
     config = {
         "input_features": [
-            category_feature(vocab_size=2, reduce_input="sum"),
+            category_feature(encoder={"type": "dense", "vocab_size": 2}, reduce_input="sum"),
             number_feature(),
         ],
         "output_features": [binary_feature(weight_regularization=None)],


Code review comments:
diff_hunk: @@ -94,16 +88,14 @@ def add_feature_data(
 
 @register_input_feature(BAG)
 class BagInputFeature(BagFeatureMixin, InputFeature):
-    encoder = "embed"
-    vocab = []
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
+    def __init__(self, input_feature_config: Union[BagInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
         if encoder_obj:
             self.encoder_obj = encoder_obj
         else:
-            self.encoder_obj = self.initialize_encoder(feature)
+            self.encoder_obj = self.initialize_encoder()
path: ludwig/features/bag_feature.py
text: @user1:
I'm not a fan of setting the encoder_config to self and then inside self.initialize_encoder() referring to it through self. I would rather pass the encoder_config explicitly to the initialize_encoder function.

@user1:
This applies to basically every input feature type

@user2:
I'm actually working on this as we speak. Changing this to a method where we pass the encoder config into the encoder initializer to create the encoder. The encoder config will then live on the encoder object. This will avoid any issues with tied weights pointing to one encoder object but an encoder config pointing to another.

@user2:
Just finished this switch so we no longer use self in the initialization of encoders and decoders. The config is directly piped in and then becomes an attribute of the `encoder_obj` / `decoder_obj`

diff_hunk: @@ -270,19 +261,17 @@ def add_feature_data(
 
 @register_input_feature(SEQUENCE)
 class SequenceInputFeature(SequenceFeatureMixin, InputFeature):
-    encoder = "parallel_cnn"
-    max_sequence_length = None
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
+    def __init__(self, input_feature_config: Union[SequenceInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
         # TODO: Potentially abstract this feature-specific attribute overwrite to a consolidated design.
-        if "vocab" in feature:
-            feature["vocab_size"] = len(feature["vocab"])
-        self.overwrite_defaults(feature)
+        self.encoder_config = input_feature_config.encoder
+        if getattr(self.encoder_config, "vocab", None):
+            self.encoder_config.vocab_size = len(self.encoder_config.vocab)
path: ludwig/features/sequence_feature.py
text: @user1:
we can revise this later, but I'm not a fan of having two parameters for vocab and vocab size around

@user2:
Agreed it is a little clunky and should probably be set more elegantly. I will create an issue on my Linear board to refactor this in a followup PR.

diff_hunk: @@ -94,16 +88,14 @@ def add_feature_data(
 
 @register_input_feature(BAG)
 class BagInputFeature(BagFeatureMixin, InputFeature):
-    encoder = "embed"
-    vocab = []
-
-    def __init__(self, feature, encoder_obj=None):
-        super().__init__(feature)
-        self.overwrite_defaults(feature)
+    def __init__(self, input_feature_config: Union[BagInputFeatureConfig, Dict], encoder_obj=None, **kwargs):
+        input_feature_config = self.load_config(input_feature_config)
+        super().__init__(input_feature_config, **kwargs)
+        self.encoder_config = input_feature_config.encoder
path: ludwig/features/bag_feature.py
text: @user1:
just a note to remind of the inconsistency of the self.encoder_config assignment in case there is a tied encoder and so encoder_obj is not None. This applies to all input feature types

@user2:
Great catch, I'm actually working on changing this right now which will address this comment and your first comment!


@user2:
This should be resolved now that the encoder and decoder configs live on the `encoder_obj` / `decoder_obj` now.

diff_hunk: @@ -479,7 +479,7 @@ def _construct_lgb_params(self) -> Tuple[dict, dict]:
                 output_params = {
                     "objective": "multiclass",
                     "metric": ["multi_logloss"],
-                    "num_class": feature.num_classes,
+                    "num_class": feature.decoder_config.num_classes,
path: ludwig/trainers/trainer_lightgbm.py
text: @user2:
I believe num_classes should likely be a property of the output feature, not the decoder, as here for the lightgbm trainer it should not be required to provide a decoder config to begin with. @user1 to doublecheck

@user3:
Awaiting @user1 reply to determine next steps!

@user1:
Agree with @user2 

@user1:
by default, GBM uses the passthrough decoder https://github.com/ludwig-ai/ludwig/blob/4405ef8745aab886e973dcdf3ad77c2048bf9f74/ludwig/utils/defaults.py#L313