Repository: ludwig-ai/ludwig

Issue:
Cannot feed value of shape for Tensor with --model_load_path for NER
**Describe the bug**
I am trying to use  pretrained model weights for initialization for building NER models using --model_load_path (defines the path of my pretrained model). Pretrained model built with 7K  records. I was trying to build another NER model and uses pretrained model weights for initialization with 1.5 K records. But I am getting Cannot feed value of shape for Tensor error while building model with below command. I am using CONLL data for training

**Command:**
ludwig experiment --data_train_csv training/training_1.csv --data_validation_csv validation/validation_1.csv --data_test_csv testing/testing_1.csv --model_load_path results/experiment_run_0/model --model_definition_file model_definition/model_definition.yaml

**Error**
 File "D:\Development_Avecto\Anaconda3\Scripts\ludwig-script.py", line 11, in <module>
    load_entry_point('ludwig==0.1.2', 'console_scripts', 'ludwig')()
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\ludwig\cli.py", line 94, in main
    CLI()
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\ludwig\cli.py", line 60, in __init__
    getattr(self, args.command)()
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\ludwig\cli.py", line 65, in experiment
    experiment.cli(sys.argv[2:])
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\ludwig\experiment.py", line 472, in cli
    experiment(**vars(args))
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\ludwig\experiment.py", line 200, in experiment
    **kwargs
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\ludwig\train.py", line 301, in full_train
    debug=debug
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\ludwig\train.py", line 461, in train
    **model_definition['training']
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\ludwig\models\model.py", line 523, in train
    is_training=True
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "D:\Development_Avecto\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1128, in _run
    str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (128, 60) for Tensor 'Tagged_Entities/Tagged_Entities_placeholder:0', which has shape '(?, 113)'

**Expected behavior**
I wanted to create the NER model using prebuilt model wiegths as initialization 



**Environment (please complete the following information):**
 - OS: Windows
 - Version:  Windows10
- Python version : 3.6.5
- Ludwig version : 0.1.2





Discussion:
@priyaJagan90 thank you for this report. My best guess so far is that the original dataset and the new one have different parameters for `max_sequence_length`.
Could you please help us replicate the error so that we can debug it?
Ideally we would need:
- yaml definition of the original model
- original data (most likely a subsample, even obfuscated or with synthetic data may work, as long as it leads to the same error)
- command for the second training importing the previous model
- the second dataset (again subsampled, obfuscated or synthesized, as long as the error happens).
Hi,

Thanks for your response. I have attached the subsample for original dataset and dataset for second run.
[Data_subset.zip](https://github.com/uber/ludwig/files/3349713/Data_subset.zip)



Resolving patch:
diff --git a/ludwig/data/preprocessing.py b/ludwig/data/preprocessing.py
--- a/ludwig/data/preprocessing.py
+++ b/ludwig/data/preprocessing.py
@@ -401,6 +401,7 @@ def preprocess_for_training(
             data_train_csv,
             data_validation_csv,
             data_test_csv,
+            train_set_metadata_json,
             skip_save_processed_input,
             preprocessing_params,
             random_seed
@@ -496,6 +497,7 @@ def _preprocess_csv_for_training(
         data_train_csv=None,
         data_validation_csv=None,
         data_test_csv=None,
+        train_set_metadata_json=None,
         skip_save_processed_input=False,
         preprocessing_params=default_preprocessing_parameters,
         random_seed=default_random_seed
@@ -507,12 +509,17 @@ def _preprocess_csv_for_training(
     :param data_train_csv:  training csv data
     :param data_validation_csv: validation csv data
     :param data_test_csv: test csv data
+    :param train_set_metadata_json: train set metadata json
     :param skip_save_processed_input: if False, the pre-processed data is saved
     as .hdf5 files in the same location as the csvs with the same names.
     :param preprocessing_params: preprocessing parameters
     :param random_seed: random seed
     :return: training, test, validation datasets, training metadata
     """
+    train_set_metadata = None
+    if train_set_metadata_json is not None:
+        train_set_metadata = load_metadata(train_set_metadata_json)
+
     if data_csv is not None:
         # Use data and ignore _train, _validation and _test.
         # Also ignore data and train set metadata needs preprocessing
@@ -525,6 +532,7 @@ def _preprocess_csv_for_training(
             data_csv,
             features,
             preprocessing_params,
+            train_set_metadata=train_set_metadata,
             random_seed=random_seed
         )
         if not skip_save_processed_input:
@@ -564,6 +572,7 @@ def _preprocess_csv_for_training(
             concatenated_df,
             features,
             preprocessing_params,
+            train_set_metadata=train_set_metadata,
             random_seed=random_seed
         )
         training_set, test_set, validation_set = split_dataset_tvt(
@@ -610,6 +619,7 @@ def _preprocess_df_for_training(
         data_train_df=None,
         data_validation_df=None,
         data_test_df=None,
+        train_set_metadata_json=None,
         preprocessing_params=default_preprocessing_parameters,
         random_seed=default_random_seed
 ):
@@ -617,6 +627,9 @@ def _preprocess_df_for_training(
     processed data as hdf5 as we don't expect users to do this as the data can
     be processed in memory
     """
+    train_set_metadata = None
+    if train_set_metadata_json is not None:
+        train_set_metadata = load_metadata(train_set_metadata_json)
 
     if data_df is not None:
         # needs preprocessing
@@ -637,6 +650,7 @@ def _preprocess_df_for_training(
         data_df,
         features,
         preprocessing_params,
+        train_set_metadata=train_set_metadata,
         random_seed=random_seed
     )
     training_set, test_set, validation_set = split_dataset_tvt(
diff --git a/ludwig/models/model.py b/ludwig/models/model.py
--- a/ludwig/models/model.py
+++ b/ludwig/models/model.py
@@ -412,6 +412,9 @@ def train(
         # ====== Setup session =======
         session = self.initialize_session(gpus, gpu_fraction)
 
+        if self.weights_save_path:
+            self.restore(session, self.weights_save_path)
+
         train_writer = None
         if is_on_master():
             if not skip_save_log:
diff --git a/ludwig/train.py b/ludwig/train.py
--- a/ludwig/train.py
+++ b/ludwig/train.py
@@ -206,6 +206,13 @@ def full_train(
         else:
             experiment_dir_name = '/'
 
+    # if model_load_path is not None, load its train_set_metadata
+    if model_load_path is not None:
+        train_set_metadata_json = os.path.join(
+            model_load_path,
+            TRAIN_SET_METADATA_FILE_NAME
+        )
+
     description_fn, training_stats_fn, model_dir = get_file_names(
         experiment_dir_name
     )


Test patch: