Repository: dadoonet/elasticsearch-beyonder (java)

Issue:
Is it possible to update mapping?
Hi there,

first: Thanks for your great work!

I am wondering whether it's possible to update mapping of an existing index. The readme is not clear about that. 

I tried start parameters, but `force` seems to always delete old index before creating new one. I supposed `merge` could be a parameter that updates given index (if possible).  
```
ElasticsearchBeyonder.start(restClient.lowLevelRest(), "elasticsearch", true, false);
```
But dosn't seem so. I tried to add a new field to my mapping json file, but it isn't created on startup (of my Spring Boot application leveraging beyonder.

So doesn't beyonder support updating mapping of existing indices? And if so: Is this intentional?

I think this is very important. Otherwise I would have to create a new index with **every** change (and re-index all data). That would be annoying and for me it would be a reason not to use beyonder, I think.

Regards and thanks in advance for an answer.

Pull request description:
Add support for update mapping
For example, let say you already have an index with the following settings and mapping:

```json
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "message": { "type": "text" },
      "foo": { "type": "text" }
    }
  }
}
```

You can provide a file named `_update_settings.json` to update your index settings
and a file named `_update_mapping.json` if you want to update an existing mapping.
Note that Elasticsearch do not allow updating all settings and mappings.

You can for example add a new field, or change the `search_analyzer` for a given field but you can not modify
the field `type`.

Considering the previous example we saw, you can create a `elasticsearch/twitter/_update_settings.json` to update the
number of replicas:

```json
{
    "number_of_replicas" : 1
}
```

And you can create `elasticsearch/twitter/_update_mapping.json`:

```json
{
  "properties": {
    "message" : {"type" : "text", "search_analyzer": "keyword" },
    "bar" : { "type" : "text" }
  }
}
```

This will change the `search_analyzer` for the `message` field and will add a new field named `bar`.
All other existing fields (like `foo` in the previous example) won't be changed.

Closes #92.

Patch:
diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml
index 72882a9..45bde11 100644
--- a/.github/workflows/pr.yml
+++ b/.github/workflows/pr.yml
@@ -3,9 +3,7 @@
 
 name: Build Pull Requests
 
-on:
-  push:
-    branches: [ master ]
+on: [pull_request]
 
 jobs:
   build:
diff --git a/README.md b/README.md
index 1096f36..bc38d82 100644
--- a/README.md
+++ b/README.md
@@ -155,17 +155,16 @@ settings for your `twitter` index.
 
 For example, create the following file `src/main/resources/elasticsearch/twitter/_settings.json` in your project:
 
-```javascript
+```json
 {
   "settings": {
     "number_of_shards": 3,
-    "number_of_replicas": 2
+    "number_of_replicas": 0
   },
   "mappings": {
     "properties": {
-      "message": {
-        "type": "text"
-      }
+      "message": { "type": "text" },
+      "foo": { "type": "text" }
     }
   }
 }
@@ -175,6 +174,36 @@ By default, Beyonder will not overwrite an index if it already exists.
 This can be overridden by setting `force` to `true` in the expanded factory method
 `ElasticsearchBeyonder.start()`.
 
+You can also provide a file named `_update_settings.json` to update your index settings 
+and a file named `_update_mapping.json` if you want to update an existing mapping. 
+Note that Elasticsearch do not allow updating all settings and mappings.
+
+You can for example add a new field, or change the `search_analyzer` for a given field but you can not modify
+the field `type`.
+
+Considering the previous example we saw, you can create a `elasticsearch/twitter/_update_settings.json` to update the
+number of replicas:
+
+```json
+{
+    "number_of_replicas" : 1
+}
+```
+
+And you can create `elasticsearch/twitter/_update_mapping.json`:
+
+```json
+{
+  "properties": {
+    "message" : {"type" : "text", "search_analyzer": "keyword" },
+    "bar" : { "type" : "text" }
+  }
+}
+```
+
+This will change the `search_analyzer` for the `message` field and will add a new field named `bar`.
+All other existing fields (like `foo` in the previous example) won't be changed.
+
 Managing templates
 ------------------
 
@@ -184,7 +213,7 @@ For example, if you planned to have indexes per year for twitter feeds (twitter2
 to define a template named `twitter_template`, you can add a file named `elasticsearch/_template/twitter_template.json`
 in your project:
 
-```javascript
+```json
 {
     "template" : "twitter*",
     "settings" : {
@@ -213,7 +242,7 @@ documents are being indexed. Please note that this feature is only supported whe
 For example, setting one fields value based on another field by using an Set Processor you an add a file named `elasticsearch/_pipeline/set_field_processor`
 in your project:
 
-```javascript
+```json
 {
   "description" : "Twitter pipeline",
   "processors" : [
diff --git a/pom.xml b/pom.xml
index 75cfcfd..8eb6607 100644
--- a/pom.xml
+++ b/pom.xml
@@ -32,8 +32,8 @@ when an elasticsearch client starts.
         <!-- For integration tests using Docker or external cluster -->
         <integ.elasticsearch.image>docker.elastic.co/elasticsearch/elasticsearch</integ.elasticsearch.image>
         <integ.elasticsearch.version>${elasticsearch.version}</integ.elasticsearch.version>
-        <tests.cluster.rest.port>9400</tests.cluster.rest.port>
-        <tests.cluster.transport.port>9500</tests.cluster.transport.port>
+        <tests.cluster.rest.port>9200</tests.cluster.rest.port>
+        <tests.cluster.transport.port>9300</tests.cluster.transport.port>
 
         <skipTests>false</skipTests>
         <skipUnitTests>${skipTests}</skipUnitTests>
@@ -167,6 +167,12 @@ when an elasticsearch client starts.
             <artifactId>commons-text</artifactId>
             <version>1.9</version>
         </dependency>
+        <dependency>
+            <groupId>commons-beanutils</groupId>
+            <artifactId>commons-beanutils</artifactId>
+            <version>1.9.4</version>
+            <scope>test</scope>
+        </dependency>
     </dependencies>
 
     <build>
diff --git a/src/main/java/fr/pilato/elasticsearch/tools/ElasticsearchBeyonder.java b/src/main/java/fr/pilato/elasticsearch/tools/ElasticsearchBeyonder.java
index ea15e92..5fac92e 100644
--- a/src/main/java/fr/pilato/elasticsearch/tools/ElasticsearchBeyonder.java
+++ b/src/main/java/fr/pilato/elasticsearch/tools/ElasticsearchBeyonder.java
@@ -32,6 +32,7 @@ import java.util.Collection;
 import java.util.List;
 
 import static fr.pilato.elasticsearch.tools.index.IndexElasticsearchUpdater.createIndex;
+import static fr.pilato.elasticsearch.tools.index.IndexElasticsearchUpdater.updateMapping;
 import static fr.pilato.elasticsearch.tools.index.IndexElasticsearchUpdater.updateSettings;
 import static fr.pilato.elasticsearch.tools.template.TemplateElasticsearchUpdater.createTemplate;
 import static fr.pilato.elasticsearch.tools.pipeline.PipelineElasticsearchUpdater.createPipeline;
@@ -122,6 +123,7 @@ public class ElasticsearchBeyonder {
 		for (String indexName : indexNames) {
 			createIndex(client, root, indexName, force);
 			updateSettings(client, root, indexName);
+			updateMapping(client, root, indexName);
 		}
 		logger.info("start done. Rock & roll!");
 	}
@@ -165,6 +167,7 @@ public class ElasticsearchBeyonder {
 		for (String indexName : indexNames) {
 			createIndex(client, root, indexName, force);
 			updateSettings(client, root, indexName);
+			updateMapping(client, root, indexName);
 		}
 		logger.info("start done. Rock & roll!");
 	}
diff --git a/src/main/java/fr/pilato/elasticsearch/tools/ResourceList.java b/src/main/java/fr/pilato/elasticsearch/tools/ResourceList.java
index b378bdc..633d46b 100644
--- a/src/main/java/fr/pilato/elasticsearch/tools/ResourceList.java
+++ b/src/main/java/fr/pilato/elasticsearch/tools/ResourceList.java
@@ -27,6 +27,7 @@ import java.io.IOException;
 import java.net.URISyntaxException;
 import java.net.URL;
 import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
 import java.util.Arrays;
 import java.util.Enumeration;
 import java.util.HashSet;
@@ -88,7 +89,7 @@ public class ResourceList {
                         // remove leading slash that is not part of the JarEntry::getName
                         .substring(1);
             Set<String> result = new HashSet<>(); //avoid duplicates in case it is a subdirectory
-            try (JarFile jar = new JarFile(URLDecoder.decode(jarPath, "UTF-8"))) {
+            try (JarFile jar = new JarFile(URLDecoder.decode(jarPath, StandardCharsets.UTF_8))) {
                 Enumeration<JarEntry> entries = jar.entries(); //gives ALL entries in jar
                 while(entries.hasMoreElements()) {
                     String name = entries.nextElement().getName();
diff --git a/src/main/java/fr/pilato/elasticsearch/tools/SettingsFinder.java b/src/main/java/fr/pilato/elasticsearch/tools/SettingsFinder.java
index e43083b..2a7da2b 100644
--- a/src/main/java/fr/pilato/elasticsearch/tools/SettingsFinder.java
+++ b/src/main/java/fr/pilato/elasticsearch/tools/SettingsFinder.java
@@ -19,23 +19,8 @@
 
 package fr.pilato.elasticsearch.tools;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.nio.file.FileVisitResult;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.nio.file.SimpleFileVisitor;
-import java.nio.file.attribute.BasicFileAttributes;
-import java.util.ArrayList;
-
-import static java.nio.file.FileVisitResult.CONTINUE;
-
 public class SettingsFinder {
 
-	private static final Logger logger = LoggerFactory.getLogger(SettingsFinder.class);
-
 	public static class Defaults {
 		/**
 		 * Default classpath dir: "elasticsearch"
@@ -45,6 +30,7 @@ public class SettingsFinder {
 		public static String JsonFileExtension = ".json";
 		public static String IndexSettingsFileName = "_settings.json";
 		public static String UpdateIndexSettingsFileName = "_update_settings.json";
+		public static String UpdateIndexMappingFileName = "_update_mapping.json";
 		public static String TemplateDir = "_template";
 		public static String PipelineDir = "_pipeline";
 
@@ -58,44 +44,4 @@ public class SettingsFinder {
 		 */
 		public static boolean ForceCreation = false;
 	}
-
-	/**
-	 * Find all types within an index
-	 * @param root dir within the classpath
-	 * @param subdir subdir name
-     * @return A list of found JSON files
-     * @deprecated Sounds like it's not used. We will remove it
-     * @throws IOException if something goes wrong
-	 */
-	@Deprecated
-	protected static ArrayList<String> findJsonFiles(Path root, String subdir) throws IOException {
-		logger.debug("Looking for json files in classpath under [{}/{}].", root, subdir);
-
-		final ArrayList<String> jsonFiles = new ArrayList<>();
-		final Path indexDir = root.resolve(subdir);
-		if (Files.exists(indexDir)) {
-			Files.walkFileTree(indexDir, new SimpleFileVisitor<Path>() {
-				@Override
-				public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
-					// We have now files. They could be type, settings, templates...
-					String jsonFile = indexDir.relativize(file).toString();
-
-					if (jsonFile.equals(Defaults.IndexSettingsFileName) ||
-							jsonFile.equals(Defaults.UpdateIndexSettingsFileName)) {
-						logger.trace("ignoring: [{}]", jsonFile);
-						return CONTINUE;
-					}
-					jsonFile = jsonFile.substring(0, jsonFile.lastIndexOf(Defaults.JsonFileExtension));
-
-					jsonFiles.add(jsonFile);
-					logger.trace("json found: [{}]", jsonFile);
-					return CONTINUE;
-				}
-			});
-		} else {
-			logger.trace("[{}] does not exist in [{}].", subdir, root);
-		}
-
-		return jsonFiles;
-	}
 }
diff --git a/src/main/java/fr/pilato/elasticsearch/tools/index/IndexElasticsearchUpdater.java b/src/main/java/fr/pilato/elasticsearch/tools/index/IndexElasticsearchUpdater.java
index 4e0e66e..8ff368b 100644
--- a/src/main/java/fr/pilato/elasticsearch/tools/index/IndexElasticsearchUpdater.java
+++ b/src/main/java/fr/pilato/elasticsearch/tools/index/IndexElasticsearchUpdater.java
@@ -163,6 +163,28 @@ public class IndexElasticsearchUpdater {
 		logger.trace("/updateIndex([{}])", index);
 	}
 
+	/**
+	 * Update mapping in Elasticsearch
+	 * @param client Elasticsearch client
+	 * @param index Index name
+	 * @param mapping Mapping if any, null if no update mapping
+	 */
+	@Deprecated
+	private static void updateMappingInElasticsearch(Client client, String index, String mapping) {
+		logger.trace("updateMapping([{}])", index);
+
+		assert client != null;
+		assert index != null;
+
+		if (mapping != null) {
+			logger.trace("Found update mapping for index [{}]: [{}]", index, mapping);
+			logger.debug("updating mapping for index [{}]", index);
+			client.admin().indices().preparePutMapping(index).setType("_doc").setSource(mapping, XContentType.JSON).get();
+		}
+
+		logger.trace("/updateMapping([{}])", index);
+	}
+
 	/**
 	 * Check if an index already exists
 	 * @param client Elasticsearch client
@@ -199,6 +221,30 @@ public class IndexElasticsearchUpdater {
 		updateIndexWithSettingsInElasticsearch(client, index, settings);
 	}
 
+
+	/**
+	 * Update index mapping in Elasticsearch. Read also _update_mapping.json if exists.
+	 * @param client Elasticsearch client
+	 * @param root dir within the classpath
+	 * @param index Index name
+	 */
+	@Deprecated
+	public static void updateMapping(Client client, String root, String index) {
+		String mapping = IndexSettingsReader.readUpdateMapping(root, index);
+		updateMappingInElasticsearch(client, index, mapping);
+	}
+
+	/**
+	 * Update index mapping in Elasticsearch. Read also _update_mapping.json if exists in default classpath dir.
+	 * @param client Elasticsearch client
+	 * @param index Index name
+	 */
+	@Deprecated
+	public static void updateMapping(Client client, String index) {
+		String mapping = IndexSettingsReader.readUpdateMapping(index);
+		updateMappingInElasticsearch(client, index, mapping);
+	}
+
 	/**
 	 * Create a new index in Elasticsearch. Read also _settings.json if exists.
 	 * @param client Elasticsearch client
@@ -321,6 +367,31 @@ public class IndexElasticsearchUpdater {
 		logger.trace("/updateIndex([{}])", index);
 	}
 
+	/**
+	 * Update mapping in Elasticsearch
+	 * @param client Elasticsearch client
+	 * @param index Index name
+	 * @param mapping Mapping if any, null if no update mapping
+	 * @throws Exception if the elasticsearch API call is failing
+	 */
+	private static void updateMappingInElasticsearch(RestClient client, String index, String mapping) throws Exception {
+		logger.trace("updateMapping([{}])", index);
+
+		assert client != null;
+		assert index != null;
+
+
+		if (mapping != null) {
+			logger.trace("Found update mapping for index [{}]: [{}]", index, mapping);
+			logger.debug("updating mapping for index [{}]", index);
+            Request request = new Request("PUT", "/" + index + "/_mapping");
+            request.setJsonEntity(mapping);
+			client.performRequest(request);
+		}
+
+		logger.trace("/updateMapping([{}])", index);
+	}
+
 	/**
 	 * Check if an index already exists
 	 * @param client Elasticsearch client
@@ -355,4 +426,27 @@ public class IndexElasticsearchUpdater {
 		String settings = IndexSettingsReader.readUpdateSettings(index);
 		updateIndexWithSettingsInElasticsearch(client, index, settings);
 	}
+
+	/**
+	 * Update index mapping in Elasticsearch. Read also _update_mapping.json if exists.
+	 * @param client Elasticsearch client
+	 * @param root dir within the classpath
+	 * @param index Index name
+	 * @throws Exception if the elasticsearch API call is failing
+	 */
+	public static void updateMapping(RestClient client, String root, String index) throws Exception {
+		String mapping = IndexSettingsReader.readUpdateMapping(root, index);
+		updateMappingInElasticsearch(client, index, mapping);
+	}
+
+	/**
+	 * Update index mapping in Elasticsearch. Read also _update_mapping.json if exists in default classpath dir.
+	 * @param client Elasticsearch client
+	 * @param index Index name
+	 * @throws Exception if the elasticsearch API call is failing
+	 */
+	public static void updateMapping(RestClient client, String index) throws Exception {
+		String mapping = IndexSettingsReader.readUpdateMapping(index);
+		updateMappingInElasticsearch(client, index, mapping);
+	}
 }
diff --git a/src/main/java/fr/pilato/elasticsearch/tools/index/IndexSettingsReader.java b/src/main/java/fr/pilato/elasticsearch/tools/index/IndexSettingsReader.java
index 4f66d66..e9b8974 100644
--- a/src/main/java/fr/pilato/elasticsearch/tools/index/IndexSettingsReader.java
+++ b/src/main/java/fr/pilato/elasticsearch/tools/index/IndexSettingsReader.java
@@ -41,7 +41,7 @@ public class IndexSettingsReader extends SettingsReader {
 	 * @param jsonFile json file to read
 	 * @return Settings
 	 */
-	public static String readSettings(String root, String index, String jsonFile) {
+	public static String readFile(String root, String index, String jsonFile) {
 		logger.trace("Reading [{}] for [{}] in [{}]...", jsonFile, index, root);
 		String settingsFile = root + "/" + index + "/" + jsonFile;
 		return readFileFromClasspath(settingsFile);
@@ -58,7 +58,7 @@ public class IndexSettingsReader extends SettingsReader {
 		if (root == null) {
 			return readSettings(index);
 		}
-		return readSettings(root, index, Defaults.IndexSettingsFileName);
+		return readFile(root, index, Defaults.IndexSettingsFileName);
 	}
 
 	/**
@@ -78,7 +78,7 @@ public class IndexSettingsReader extends SettingsReader {
      * @return Update Settings
 	 */
 	public static String readUpdateSettings(String root, String index) {
-		return readSettings(root, index, Defaults.UpdateIndexSettingsFileName);
+		return readFile(root, index, Defaults.UpdateIndexSettingsFileName);
 	}
 
 	/**
@@ -89,4 +89,23 @@ public class IndexSettingsReader extends SettingsReader {
 	public static String readUpdateSettings(String index) {
 		return readUpdateSettings(Defaults.ConfigDir, index);
 	}
+
+	/**
+	 * Read index mapping
+	 * @param root dir within the classpath
+	 * @param index index name
+     * @return Update Mapping
+	 */
+	public static String readUpdateMapping(String root, String index) {
+		return readFile(root, index, Defaults.UpdateIndexMappingFileName);
+	}
+
+	/**
+	 * Read index mapping in default classpath dir
+	 * @param index index name
+     * @return Update Mapping
+     */
+	public static String readUpdateMapping(String index) {
+		return readUpdateMapping(Defaults.ConfigDir, index);
+	}
 }


Test patch:
diff --git a/src/test/java/fr/pilato/elasticsearch/tools/AbstractBeyonderTest.java b/src/test/java/fr/pilato/elasticsearch/tools/AbstractBeyonderTest.java
index 9ef8ab3..1d97a52 100644
--- a/src/test/java/fr/pilato/elasticsearch/tools/AbstractBeyonderTest.java
+++ b/src/test/java/fr/pilato/elasticsearch/tools/AbstractBeyonderTest.java
@@ -19,7 +19,6 @@
 
 package fr.pilato.elasticsearch.tools;
 
-import fr.pilato.elasticsearch.tools.index.IndexSettingsReader;
 import org.apache.http.HttpHost;
 import org.apache.http.auth.AuthScope;
 import org.apache.http.auth.UsernamePasswordCredentials;
@@ -33,22 +32,22 @@ import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import java.io.ByteArrayInputStream;
 import java.io.IOException;
 import java.net.ConnectException;
 import java.util.List;
 import java.util.Map;
 
 import static java.util.Collections.singletonList;
-import static org.hamcrest.Matchers.*;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.not;
 import static org.junit.Assume.assumeThat;
 
 public abstract class AbstractBeyonderTest {
 
     static final Logger logger = LoggerFactory.getLogger(AbstractBeyonderTest.class);
 
-    private final static String DEFAULT_TEST_CLUSTER = "http://127.0.0.1:9400";
-    private final static Integer DEFAULT_TEST_CLUSTER_TRANSPORT_PORT = 9500;
+    private final static String DEFAULT_TEST_CLUSTER = "http://127.0.0.1:9200";
+    private final static Integer DEFAULT_TEST_CLUSTER_TRANSPORT_PORT = 9300;
 
     final static String testCluster = System.getProperty("tests.cluster", DEFAULT_TEST_CLUSTER);
     final static String testClusterUser = System.getProperty("tests.cluster.user");
@@ -118,24 +117,6 @@ public abstract class AbstractBeyonderTest {
                 null);
     }
 
-    @Test
-    public void testVariableReplacement() throws Exception {
-
-      // given: A settings json with a variable that should be replaced.
-      //        And an environment variable with matching name (set in configuration of maven-surefire-plugin).
-      String folder = "models/variablereplacement";
-      String indexName = "twitter";
-
-      // when: this settings file is read
-      String settings = IndexSettingsReader.readSettings(folder, indexName);
-      Map<String, Object> settingsMap = JsonUtil.asMap(new ByteArrayInputStream(settings.getBytes()));
-
-      // then: the variables got replaced by environment variables of the same name
-      String numberOfReplicas = (String) ((Map) settingsMap.get("settings")).get("number_of_replicas");
-      assumeThat(numberOfReplicas, equalTo("2"));
-
-    }
-
     @Test
     public void testSettingsAnalyzer() throws Exception {
         // Custom settings (analyzer)
@@ -160,22 +141,10 @@ public abstract class AbstractBeyonderTest {
                 singletonList("twitter_template"));
     }
 
-    @Test
-    public void testUpdateSettings() throws Exception {
-        // 1 _update_settings
-        testBeyonder("models/update-settings/step1",
-                singletonList("twitter"),
-                null);
-        testBeyonder("models/update-settings/step2",
-                singletonList("twitter"),
-                null);
-    }
-
     @Test
     public void testWrongClasspathDir() throws Exception {
         testBeyonder("models/bad-classpath-7/doesnotexist",
                 null,
                 null);
     }
-
 }
diff --git a/src/test/java/fr/pilato/elasticsearch/tools/BeyonderRestIT.java b/src/test/java/fr/pilato/elasticsearch/tools/BeyonderRestIT.java
index e74b5d3..b9a36b0 100644
--- a/src/test/java/fr/pilato/elasticsearch/tools/BeyonderRestIT.java
+++ b/src/test/java/fr/pilato/elasticsearch/tools/BeyonderRestIT.java
@@ -22,7 +22,7 @@ package fr.pilato.elasticsearch.tools;
 import fr.pilato.elasticsearch.tools.alias.AliasElasticsearchUpdater;
 import fr.pilato.elasticsearch.tools.index.IndexElasticsearchUpdater;
 import fr.pilato.elasticsearch.tools.pipeline.PipelineElasticsearchUpdater;
-
+import org.apache.commons.beanutils.BeanUtils;
 import org.apache.commons.io.IOUtils;
 import org.apache.http.HttpEntity;
 import org.elasticsearch.client.Request;
@@ -43,10 +43,14 @@ import java.util.Map;
 import static fr.pilato.elasticsearch.tools.JsonUtil.asMap;
 import static fr.pilato.elasticsearch.tools.index.IndexElasticsearchUpdater.isIndexExist;
 import static fr.pilato.elasticsearch.tools.template.TemplateElasticsearchUpdater.isTemplateExist;
+import static java.util.Collections.singletonList;
 import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.hasKey;
 import static org.hamcrest.Matchers.is;
 import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.notNullValue;
+import static org.hamcrest.Matchers.nullValue;
 import static org.junit.Assume.assumeNoException;
 
 public class BeyonderRestIT extends AbstractBeyonderTest {
@@ -151,6 +155,62 @@ public class BeyonderRestIT extends AbstractBeyonderTest {
 		assertThat(PipelineElasticsearchUpdater.isPipelineExist(client, "twitter_pipeline"), is(true));
 	}
 
+    @Test
+    public void testUpdateSettings() throws Exception {
+        // 1 _settings
+        {
+            testBeyonder("models/update-settings/step1",
+                    singletonList("twitter"),
+                    null);
+            Map<String, Object> oldSettings = asMap(client.performRequest(new Request("GET", "/twitter/_settings")));
+            String numberOfReplicas = BeanUtils.getProperty(oldSettings, "twitter.settings.index.number_of_replicas");
+            assertThat(numberOfReplicas, equalTo("0"));
+        }
+
+        // 2 _update_settings
+        {
+            testBeyonder("models/update-settings/step2",
+                    singletonList("twitter"),
+                    null);
+            Map<String, Object> settings = asMap(client.performRequest(new Request("GET", "/twitter/_settings")));
+            String numberOfReplicas = BeanUtils.getProperty(settings, "twitter.settings.index.number_of_replicas");
+            assertThat(numberOfReplicas, equalTo("1"));
+        }
+    }
+
+    @Test
+    public void testUpdateMapping() throws Exception {
+        // 1 _settings
+        {
+            testBeyonder("models/update-mapping/step1",
+                    singletonList("twitter"),
+                    null);
+
+            Map<String, Object> mapping = asMap(client.performRequest(new Request("GET", "/twitter/_mappings")));
+            String bar = BeanUtils.getProperty(mapping, "twitter.mappings.properties.bar");
+            String foo = BeanUtils.getProperty(mapping, "twitter.mappings.properties.foo");
+            String message = BeanUtils.getProperty(mapping, "twitter.mappings.properties.message.search_analyzer");
+            assertThat(bar, nullValue());
+            assertThat(foo, notNullValue());
+            assertThat(message, nullValue());
+        }
+
+        // 2 _update_mapping
+        {
+            testBeyonder("models/update-mapping/step2",
+                    singletonList("twitter"),
+                    null);
+
+            Map<String, Object> mapping = asMap(client.performRequest(new Request("GET", "/twitter/_mappings")));
+            String bar = BeanUtils.getProperty(mapping, "twitter.mappings.properties.bar");
+            String foo = BeanUtils.getProperty(mapping, "twitter.mappings.properties.foo");
+            String message = BeanUtils.getProperty(mapping, "twitter.mappings.properties.message.search_analyzer");
+            assertThat(bar, notNullValue());
+            assertThat(foo, notNullValue());
+            assertThat(message, notNullValue());
+        }
+    }
+
     private String getMapping(String indexName) throws IOException {
         HttpEntity response = client.performRequest(new Request("GET", indexName + "/_mapping")).getEntity();
         ByteArrayOutputStream out = new ByteArrayOutputStream(Math.toIntExact(response.getContentLength()));
diff --git a/src/test/java/fr/pilato/elasticsearch/tools/BeyonderTransportIT.java b/src/test/java/fr/pilato/elasticsearch/tools/BeyonderTransportIT.java
index 5495651..a3e0781 100644
--- a/src/test/java/fr/pilato/elasticsearch/tools/BeyonderTransportIT.java
+++ b/src/test/java/fr/pilato/elasticsearch/tools/BeyonderTransportIT.java
@@ -19,6 +19,8 @@
 
 package fr.pilato.elasticsearch.tools;
 
+import org.apache.commons.beanutils.BeanUtils;
+import org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.client.RestClient;
 import org.elasticsearch.client.transport.NoNodeAvailableException;
@@ -35,11 +37,16 @@ import org.slf4j.LoggerFactory;
 import java.io.IOException;
 import java.net.InetSocketAddress;
 import java.util.List;
+import java.util.Map;
 
 import static fr.pilato.elasticsearch.tools.index.IndexElasticsearchUpdater.isIndexExist;
 import static fr.pilato.elasticsearch.tools.template.TemplateElasticsearchUpdater.isTemplateExist;
+import static java.util.Collections.singletonList;
 import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.notNullValue;
+import static org.hamcrest.Matchers.nullValue;
 import static org.junit.Assume.assumeNoException;
 import static org.junit.Assume.assumeTrue;
 
@@ -92,6 +99,61 @@ public class BeyonderTransportIT extends AbstractBeyonderTest {
                 "be provided)", false);
     }
 
+    @Test
+    public void testUpdateSettings() throws Exception {
+        // 1 _settings
+        {
+            testBeyonder("models/update-settings/step1",
+                    singletonList("twitter"),
+                    null);
+            GetSettingsResponse settings = client.admin().indices().prepareGetSettings("twitter").get();
+            String numberOfReplicas = settings.getSetting("twitter", "index.number_of_replicas");
+            assertThat(numberOfReplicas, equalTo("0"));
+        }
+
+        // 2 _update_settings
+        {
+            testBeyonder("models/update-settings/step2",
+                    singletonList("twitter"),
+                    null);
+            GetSettingsResponse settings = client.admin().indices().prepareGetSettings("twitter").get();
+            String numberOfReplicas = settings.getSetting("twitter", "index.number_of_replicas");
+            assertThat(numberOfReplicas, equalTo("1"));
+        }
+    }
+
+    @Test
+    public void testUpdateMapping() throws Exception {
+        // 1 _settings
+        {
+            testBeyonder("models/update-mapping/step1",
+                    singletonList("twitter"),
+                    null);
+            Map<String, Object> properties = client.admin().indices().prepareGetMappings("twitter").get().getMappings().get("twitter").get("_doc").getSourceAsMap();
+            String bar = BeanUtils.getProperty(properties, "properties.bar");
+            String foo = BeanUtils.getProperty(properties, "properties.foo");
+            String message = BeanUtils.getProperty(properties, "properties.message.search_analyzer");
+            assertThat(bar, nullValue());
+            assertThat(foo, notNullValue());
+            assertThat(message, nullValue());
+        }
+
+        // 2 _update_mapping
+        {
+            testBeyonder("models/update-mapping/step2",
+                    singletonList("twitter"),
+                    null);
+
+            Map<String, Object> properties = client.admin().indices().prepareGetMappings("twitter").get().getMappings().get("twitter").get("_doc").getSourceAsMap();
+            String bar = BeanUtils.getProperty(properties, "properties.bar");
+            String foo = BeanUtils.getProperty(properties, "properties.foo");
+            String message = BeanUtils.getProperty(properties, "properties.message.search_analyzer");
+            assertThat(bar, notNullValue());
+            assertThat(foo, notNullValue());
+            assertThat(message, notNullValue());
+        }
+    }
+
     protected void testBeyonder(String root,
                                 List<String> indices,
                                 List<String> templates) throws Exception {
diff --git a/src/test/java/fr/pilato/elasticsearch/tools/BeyonderUnitTest.java b/src/test/java/fr/pilato/elasticsearch/tools/BeyonderUnitTest.java
index 6daa115..0e26347 100644
--- a/src/test/java/fr/pilato/elasticsearch/tools/BeyonderUnitTest.java
+++ b/src/test/java/fr/pilato/elasticsearch/tools/BeyonderUnitTest.java
@@ -22,14 +22,21 @@ package fr.pilato.elasticsearch.tools;
 import fr.pilato.elasticsearch.tools.index.IndexSettingsReader;
 import fr.pilato.elasticsearch.tools.template.TemplateFinder;
 import fr.pilato.elasticsearch.tools.template.TemplateSettingsReader;
+import org.junit.Test;
 
+import java.io.ByteArrayInputStream;
 import java.io.IOException;
 import java.net.URISyntaxException;
 import java.util.List;
+import java.util.Map;
 
 import static fr.pilato.elasticsearch.tools.index.IndexFinder.findIndexNames;
+import static java.util.Collections.singletonList;
 import static org.hamcrest.MatcherAssert.assertThat;
-import static org.hamcrest.Matchers.*;
+import static org.hamcrest.Matchers.emptyIterable;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.is;
 
 public class BeyonderUnitTest extends AbstractBeyonderTest {
 
@@ -78,4 +85,48 @@ public class BeyonderUnitTest extends AbstractBeyonderTest {
             assertThat(templateNames, emptyIterable());
         }
     }
+
+    @Test
+    public void testVariableReplacement() throws Exception {
+
+        // given: A settings json with a variable that should be replaced.
+        //        And an environment variable with matching name (set in configuration of maven-surefire-plugin).
+        String folder = "models/variablereplacement";
+        String indexName = "twitter";
+
+        // when: this settings file is read
+        String settings = IndexSettingsReader.readSettings(folder, indexName);
+        Map<String, Object> settingsMap = JsonUtil.asMap(new ByteArrayInputStream(settings.getBytes()));
+
+        // then: the variables got replaced by environment variables of the same name
+        String numberOfReplicas = (String) ((Map) settingsMap.get("settings")).get("number_of_replicas");
+        assertThat(numberOfReplicas, equalTo("2"));
+
+    }
+
+    @Test
+    public void testUpdateMapping() throws Exception {
+        // 1 _settings
+        testBeyonder("models/update-mapping/step1",
+                singletonList("twitter"),
+                null);
+
+        // 2 _update_mapping
+        testBeyonder("models/update-mapping/step2",
+                singletonList("twitter"),
+                null);
+    }
+
+    @Test
+    public void testUpdateSettings() throws Exception {
+        // 1 _settings
+        testBeyonder("models/update-settings/step1",
+                singletonList("twitter"),
+                null);
+
+        // 2 _update_settings
+        testBeyonder("models/update-settings/step2",
+                singletonList("twitter"),
+                null);
+    }
 }
diff --git a/src/test/java/fr/pilato/elasticsearch/tools/JsonUtil.java b/src/test/java/fr/pilato/elasticsearch/tools/JsonUtil.java
index 7483a3e..d2f8913 100644
--- a/src/test/java/fr/pilato/elasticsearch/tools/JsonUtil.java
+++ b/src/test/java/fr/pilato/elasticsearch/tools/JsonUtil.java
@@ -44,7 +44,7 @@ public class JsonUtil {
 
     public static Map<String, Object> asMap(InputStream stream) {
         try {
-            return mapper.readValue(stream, new TypeReference<Map<String, Object>>(){});
+            return mapper.readValue(stream, new TypeReference<>() {});
         } catch (IOException e) {
             throw new RuntimeException(e);
         }
diff --git a/src/test/resources/models/update-mapping/step1/twitter/_settings.json b/src/test/resources/models/update-mapping/step1/twitter/_settings.json
new file mode 100644
index 0000000..3bd3322
--- /dev/null
+++ b/src/test/resources/models/update-mapping/step1/twitter/_settings.json
@@ -0,0 +1,8 @@
+{
+  "mappings" : {
+    "properties" : {
+      "message" : {"type" : "text", "store" : true, "analyzer" : "simple" },
+      "foo" : { "type" : "text" }
+    }
+  }
+}
diff --git a/src/test/resources/models/update-mapping/step2/twitter/_update_mapping.json b/src/test/resources/models/update-mapping/step2/twitter/_update_mapping.json
new file mode 100644
index 0000000..cbd8e00
--- /dev/null
+++ b/src/test/resources/models/update-mapping/step2/twitter/_update_mapping.json
@@ -0,0 +1,6 @@
+{
+    "properties": {
+        "message" : {"type" : "text", "store" : true, "analyzer" : "simple", "search_analyzer": "keyword" },
+        "bar" : { "type" : "text" }
+    }
+}
diff --git a/src/test/resources/transport/models/update-mapping/step1/twitter/_settings.json b/src/test/resources/transport/models/update-mapping/step1/twitter/_settings.json
new file mode 100644
index 0000000..e8898bc
--- /dev/null
+++ b/src/test/resources/transport/models/update-mapping/step1/twitter/_settings.json
@@ -0,0 +1,10 @@
+{
+  "mappings" : {
+    "_doc": {
+      "properties" : {
+        "message" : {"type" : "text", "store" : true, "analyzer" : "simple" },
+        "foo" : { "type" : "text" }
+      }
+    }
+  }
+}
diff --git a/src/test/resources/transport/models/update-mapping/step2/twitter/_update_mapping.json b/src/test/resources/transport/models/update-mapping/step2/twitter/_update_mapping.json
new file mode 100644
index 0000000..cbd8e00
--- /dev/null
+++ b/src/test/resources/transport/models/update-mapping/step2/twitter/_update_mapping.json
@@ -0,0 +1,6 @@
+{
+    "properties": {
+        "message" : {"type" : "text", "store" : true, "analyzer" : "simple", "search_analyzer": "keyword" },
+        "bar" : { "type" : "text" }
+    }
+}