Repository: LangStream/langstream (language: python)

Issue:
Proposal for tenant resources guardrails
## Requirements
- Add a limit to the number of resource that a tenant can create

Currently any agent can specify parallelism and unit size. This is great because there's an abstraction over the actual resources which is easier to control.

The guardrails are meant to reject new application deployment (or updates) if the total of existing applications size would exceed the limit. Since the control plane can be replicated and also multiple requests might be sent in parallel, there's a need for a locking system or solution.

From the DX point of view, it's important to get the application being rejected at deploy time (control plane) and not at runtime. (checking the app status).

The default value initially will be static and the same for each tenant, configurable in the configmap. 
On top of that, the value will modifiable for each tenant by changing the tenant metadata.

A change to the guardrail value must NOT impact existing applications. It's meant to be applied only on new applications (or updates of existing's)

## Proposal
The solution is based on the two verification steps.
 
### First check (control plane)
The first check happens in the control plane when the user deploy/updates an application. This check is best-effort - the control plane will check the current applications and perform calculation.
This check is lightweight and it already covers most of the cases, when the user sends only one request at the time. 
If the user (or more users in the same tenant) sends more than one request at the time, this check could be deceived, since the control plane doesn't keep a consistent state locally about all the applications (also control plane have multiple replicas)

This first check is aimed to improve user experience with fast feedbacks but it doesn't guarantee resource usage to stay under the threshold.

## Second check (deployer)
The deployer is single machine. Even with more than one replicas, there's only one leader. 
The deployer then can implement a resource limit service keeping a local count of resource usage by tenant.
The deployer then will accept or reject the application. Note that the user won't receive the feedback at the request-time.
Once the deployer rejected an application, the status will be moved to ERROR_DEPLOYING with an appropriate message.

Since the limit is dynamic and also other applications could be deleted/downscaled, the deployer will continue to perform the check periodically.

This second check guarantees consistency and well applied resource guardrails. 


Discussion / hints:
I think we should build in a per-tenant override from the beginning. As soon as you put in guardrails like this, the first request will be for an exception for a specific tenant.

Fix patch:
diff --git a/langstream-api/src/main/java/ai/langstream/api/storage/ApplicationStore.java b/langstream-api/src/main/java/ai/langstream/api/storage/ApplicationStore.java
index 04490be01..5f25ca4bb 100644
--- a/langstream-api/src/main/java/ai/langstream/api/storage/ApplicationStore.java
+++ b/langstream-api/src/main/java/ai/langstream/api/storage/ApplicationStore.java
@@ -30,6 +30,8 @@ public interface ApplicationStore extends GenericStore {
 
     void onTenantCreated(String tenant);
 
+    void onTenantUpdated(String tenant);
+
     void onTenantDeleted(String tenant);
 
     void put(
diff --git a/langstream-api/src/main/java/ai/langstream/api/storage/GlobalMetadataStore.java b/langstream-api/src/main/java/ai/langstream/api/storage/GlobalMetadataStore.java
index 28759596a..3c2ad5642 100644
--- a/langstream-api/src/main/java/ai/langstream/api/storage/GlobalMetadataStore.java
+++ b/langstream-api/src/main/java/ai/langstream/api/storage/GlobalMetadataStore.java
@@ -18,6 +18,7 @@
 import java.util.LinkedHashMap;
 
 public interface GlobalMetadataStore extends GenericStore {
+    String TENANT_KEY_PREFIX = "t-";
 
     void put(String key, String value);
 
diff --git a/langstream-api/src/main/java/ai/langstream/api/webservice/tenant/CreateTenantRequest.java b/langstream-api/src/main/java/ai/langstream/api/webservice/tenant/CreateTenantRequest.java
new file mode 100644
index 000000000..cf2e45a31
--- /dev/null
+++ b/langstream-api/src/main/java/ai/langstream/api/webservice/tenant/CreateTenantRequest.java
@@ -0,0 +1,30 @@
+/*
+ * Copyright DataStax, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package ai.langstream.api.webservice.tenant;
+
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+@Builder
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+public class CreateTenantRequest {
+
+    private Integer maxTotalResourceUnits;
+}
diff --git a/langstream-api/src/main/java/ai/langstream/api/model/TenantConfiguration.java b/langstream-api/src/main/java/ai/langstream/api/webservice/tenant/TenantConfiguration.java
similarity index 90%
rename from langstream-api/src/main/java/ai/langstream/api/model/TenantConfiguration.java
rename to langstream-api/src/main/java/ai/langstream/api/webservice/tenant/TenantConfiguration.java
index f6a933ad8..09b5463b2 100644
--- a/langstream-api/src/main/java/ai/langstream/api/model/TenantConfiguration.java
+++ b/langstream-api/src/main/java/ai/langstream/api/webservice/tenant/TenantConfiguration.java
@@ -13,7 +13,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package ai.langstream.api.model;
+package ai.langstream.api.webservice.tenant;
 
 import lombok.AllArgsConstructor;
 import lombok.Builder;
@@ -26,4 +26,5 @@
 @AllArgsConstructor
 public class TenantConfiguration {
     private String name;
+    private int maxTotalResourceUnits;
 }
diff --git a/langstream-api/src/main/java/ai/langstream/api/webservice/tenant/UpdateTenantRequest.java b/langstream-api/src/main/java/ai/langstream/api/webservice/tenant/UpdateTenantRequest.java
new file mode 100644
index 000000000..75ac32d8c
--- /dev/null
+++ b/langstream-api/src/main/java/ai/langstream/api/webservice/tenant/UpdateTenantRequest.java
@@ -0,0 +1,30 @@
+/*
+ * Copyright DataStax, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package ai.langstream.api.webservice.tenant;
+
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+@Builder
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+public class UpdateTenantRequest {
+
+    private Integer maxTotalResourceUnits;
+}
diff --git a/langstream-cli/src/main/java/ai/langstream/cli/commands/BaseCmd.java b/langstream-cli/src/main/java/ai/langstream/cli/commands/BaseCmd.java
index 72f0957cf..80c6c9a6a 100644
--- a/langstream-cli/src/main/java/ai/langstream/cli/commands/BaseCmd.java
+++ b/langstream-cli/src/main/java/ai/langstream/cli/commands/BaseCmd.java
@@ -21,6 +21,7 @@
 import ai.langstream.cli.LangStreamCLIConfig;
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectWriter;
 import com.fasterxml.jackson.databind.SerializationFeature;
 import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
 import java.io.File;
@@ -53,6 +54,8 @@ public enum Formats {
                     .enable(SerializationFeature.INDENT_OUTPUT)
                     .enable(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS);
 
+    protected static final ObjectWriter jsonBodyWriter = new ObjectMapper().writer();
+
     @CommandLine.Spec protected CommandLine.Model.CommandSpec command;
     private AdminClient client;
     private LangStreamCLIConfig config;
diff --git a/langstream-cli/src/main/java/ai/langstream/cli/commands/RootTenantCmd.java b/langstream-cli/src/main/java/ai/langstream/cli/commands/RootTenantCmd.java
index 4020a8e8c..ba4b24e59 100644
--- a/langstream-cli/src/main/java/ai/langstream/cli/commands/RootTenantCmd.java
+++ b/langstream-cli/src/main/java/ai/langstream/cli/commands/RootTenantCmd.java
@@ -15,10 +15,12 @@
  */
 package ai.langstream.cli.commands;
 
+import ai.langstream.cli.commands.tenants.CreateTenantCmd;
 import ai.langstream.cli.commands.tenants.DeleteTenantCmd;
 import ai.langstream.cli.commands.tenants.GetTenantCmd;
 import ai.langstream.cli.commands.tenants.ListTenantCmd;
 import ai.langstream.cli.commands.tenants.PutTenantCmd;
+import ai.langstream.cli.commands.tenants.UpdateTenantCmd;
 import lombok.Getter;
 import picocli.CommandLine;
 
@@ -30,7 +32,9 @@
             PutTenantCmd.class,
             DeleteTenantCmd.class,
             ListTenantCmd.class,
-            GetTenantCmd.class
+            GetTenantCmd.class,
+            CreateTenantCmd.class,
+            UpdateTenantCmd.class
         })
 @Getter
 public class RootTenantCmd {
diff --git a/langstream-cli/src/main/java/ai/langstream/cli/commands/tenants/CreateTenantCmd.java b/langstream-cli/src/main/java/ai/langstream/cli/commands/tenants/CreateTenantCmd.java
new file mode 100644
index 000000000..c3452929e
--- /dev/null
+++ b/langstream-cli/src/main/java/ai/langstream/cli/commands/tenants/CreateTenantCmd.java
@@ -0,0 +1,53 @@
+/*
+ * Copyright DataStax, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package ai.langstream.cli.commands.tenants;
+
+import ai.langstream.api.webservice.tenant.CreateTenantRequest;
+import java.net.http.HttpRequest;
+import lombok.SneakyThrows;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+        name = "create",
+        mixinStandardHelpOptions = true,
+        description = "Create a new tenant")
+public class CreateTenantCmd extends BaseTenantCmd {
+
+    @CommandLine.Parameters(description = "Name of the tenant")
+    private String name;
+
+    @CommandLine.Option(
+            names = {"--max-total-resource-units"},
+            description = "Max application's units for the tenant.")
+    private Integer maxUnits;
+
+    @Override
+    @SneakyThrows
+    public void run() {
+        final CreateTenantRequest request =
+                CreateTenantRequest.builder().maxTotalResourceUnits(maxUnits).build();
+        final String body = jsonBodyWriter.writeValueAsString(request);
+
+        final HttpRequest httpRequest =
+                getClient()
+                        .newPost(
+                                pathForTenant(name),
+                                "application/json",
+                                HttpRequest.BodyPublishers.ofString(body));
+        getClient().http(httpRequest);
+        log("tenant %s created".formatted(name));
+    }
+}
diff --git a/langstream-cli/src/main/java/ai/langstream/cli/commands/tenants/UpdateTenantCmd.java b/langstream-cli/src/main/java/ai/langstream/cli/commands/tenants/UpdateTenantCmd.java
new file mode 100644
index 000000000..0fc157b12
--- /dev/null
+++ b/langstream-cli/src/main/java/ai/langstream/cli/commands/tenants/UpdateTenantCmd.java
@@ -0,0 +1,53 @@
+/*
+ * Copyright DataStax, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package ai.langstream.cli.commands.tenants;
+
+import ai.langstream.api.webservice.tenant.UpdateTenantRequest;
+import java.net.http.HttpRequest;
+import lombok.SneakyThrows;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+        name = "update",
+        mixinStandardHelpOptions = true,
+        description = "Update an existing tenant")
+public class UpdateTenantCmd extends BaseTenantCmd {
+
+    @CommandLine.Parameters(description = "Name of the tenant")
+    private String name;
+
+    @CommandLine.Option(
+            names = {"--max-total-resource-units"},
+            description = "Max application's units for the tenant.")
+    private Integer maxUnits;
+
+    @Override
+    @SneakyThrows
+    public void run() {
+        final UpdateTenantRequest request =
+                UpdateTenantRequest.builder().maxTotalResourceUnits(maxUnits).build();
+        final String body = jsonBodyWriter.writeValueAsString(request);
+
+        final HttpRequest httpRequest =
+                getClient()
+                        .newPatch(
+                                pathForTenant(name),
+                                "application/json",
+                                HttpRequest.BodyPublishers.ofString(body));
+        getClient().http(httpRequest);
+        log("tenant %s updated".formatted(name));
+    }
+}
diff --git a/langstream-core/src/main/java/ai/langstream/impl/storage/GlobalMetadataStoreManager.java b/langstream-core/src/main/java/ai/langstream/impl/storage/GlobalMetadataStoreManager.java
index e0b6a6909..53d2d4d17 100644
--- a/langstream-core/src/main/java/ai/langstream/impl/storage/GlobalMetadataStoreManager.java
+++ b/langstream-core/src/main/java/ai/langstream/impl/storage/GlobalMetadataStoreManager.java
@@ -15,20 +15,27 @@
  */
 package ai.langstream.impl.storage;
 
-import ai.langstream.api.model.TenantConfiguration;
 import ai.langstream.api.storage.ApplicationStore;
 import ai.langstream.api.storage.GlobalMetadataStore;
+import ai.langstream.api.webservice.tenant.CreateTenantRequest;
+import ai.langstream.api.webservice.tenant.TenantConfiguration;
+import ai.langstream.api.webservice.tenant.UpdateTenantRequest;
+import ai.langstream.impl.storage.tenants.TenantException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import java.util.Map;
 import java.util.stream.Collectors;
 import lombok.AllArgsConstructor;
 import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
 
 @AllArgsConstructor
+@Slf4j
 public class GlobalMetadataStoreManager {
 
-    private static final ObjectMapper mapper = new ObjectMapper();
-    protected static final String TENANT_KEY_PREFIX = "t-";
+    private static final ObjectMapper mapper =
+            new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+
     private final GlobalMetadataStore globalMetadataStore;
     private final ApplicationStore applicationStore;
 
@@ -40,11 +47,70 @@ public void syncTenantConfiguration(String tenant) {
         applicationStore.onTenantCreated(tenant);
     }
 
+    @SneakyThrows
+    public void createTenant(String tenant, CreateTenantRequest request) throws TenantException {
+        final String key = keyedTenantName(tenant);
+        if (globalMetadataStore.get(key) != null) {
+            throw new TenantException(
+                    "Tenant " + tenant + " already exists", TenantException.Type.AlreadyExists);
+        }
+        final TenantConfiguration configuration = createConfiguration(tenant, request);
+        log.info("Creating tenant {} with configuration {}", tenant, configuration);
+
+        globalMetadataStore.put(key, mapper.writeValueAsString(configuration));
+        applicationStore.onTenantCreated(tenant);
+    }
+
+    private TenantConfiguration createConfiguration(String tenant, CreateTenantRequest request) {
+        final TenantConfiguration.TenantConfigurationBuilder builder =
+                TenantConfiguration.builder().name(tenant);
+        final Integer maxTotalResourceUnits = request.getMaxTotalResourceUnits();
+        if (maxTotalResourceUnits != null && maxTotalResourceUnits < 0) {
+            throw new IllegalArgumentException("maxTotalResourceUnits must be positive");
+        }
+        builder.maxTotalResourceUnits(
+                maxTotalResourceUnits == null ? 0 : maxTotalResourceUnits.intValue());
+
+        final TenantConfiguration configuration = builder.build();
+        return configuration;
+    }
+
+    @SneakyThrows
+    public void updateTenant(String tenant, UpdateTenantRequest request) throws TenantException {
+        final String key = keyedTenantName(tenant);
+        final String before = globalMetadataStore.get(key);
+        if (before == null) {
+            throw new TenantException(
+                    "Tenant " + tenant + " not found", TenantException.Type.NotFound);
+        }
+        final TenantConfiguration mergeConfig = mapper.readValue(before, TenantConfiguration.class);
+        mergeConfiguration(request, mergeConfig);
+
+        log.info("Updating tenant {} with configuration {}", tenant, mergeConfig);
+
+        globalMetadataStore.put(key, mapper.writeValueAsString(mergeConfig));
+        applicationStore.onTenantUpdated(tenant);
+    }
+
+    private void mergeConfiguration(UpdateTenantRequest request, TenantConfiguration mergeConfig) {
+        final Integer maxTotalResourceUnits = request.getMaxTotalResourceUnits();
+        if (maxTotalResourceUnits != null && maxTotalResourceUnits < 0) {
+            throw new IllegalArgumentException("maxTotalResourceUnits must be positive");
+        }
+        mergeConfig.setMaxTotalResourceUnits(
+                maxTotalResourceUnits != null ? maxTotalResourceUnits : 0);
+    }
+
     @SneakyThrows
     public void putTenant(String tenant, TenantConfiguration tenantConfiguration) {
         final String key = keyedTenantName(tenant);
+        final String before = globalMetadataStore.get(key);
         globalMetadataStore.put(key, mapper.writeValueAsString(tenantConfiguration));
-        applicationStore.onTenantCreated(tenant);
+        if (before != null) {
+            applicationStore.onTenantUpdated(tenant);
+        } else {
+            applicationStore.onTenantCreated(tenant);
+        }
     }
 
     public TenantConfiguration getTenant(String tenant) {
@@ -71,14 +137,18 @@ public void deleteTenant(String tenant) {
     @SneakyThrows
     public Map<String, TenantConfiguration> listTenants() {
         return globalMetadataStore.list().entrySet().stream()
-                .filter(e -> e.getKey().startsWith(TENANT_KEY_PREFIX))
+                .filter(e -> e.getKey().startsWith(GlobalMetadataStore.TENANT_KEY_PREFIX))
                 .collect(
                         Collectors.toMap(
-                                e -> e.getKey().substring(TENANT_KEY_PREFIX.length()),
+                                e ->
+                                        e.getKey()
+                                                .substring(
+                                                        GlobalMetadataStore.TENANT_KEY_PREFIX
+                                                                .length()),
                                 e -> parseTenantConfiguration(e.getValue())));
     }
 
-    private static String keyedTenantName(String tenant) {
-        return TENANT_KEY_PREFIX + tenant;
+    public static String keyedTenantName(String tenant) {
+        return GlobalMetadataStore.TENANT_KEY_PREFIX + tenant;
     }
 }
diff --git a/langstream-core/src/main/java/ai/langstream/impl/storage/tenants/TenantException.java b/langstream-core/src/main/java/ai/langstream/impl/storage/tenants/TenantException.java
new file mode 100644
index 000000000..41325cdd7
--- /dev/null
+++ b/langstream-core/src/main/java/ai/langstream/impl/storage/tenants/TenantException.java
@@ -0,0 +1,33 @@
+/*
+ * Copyright DataStax, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package ai.langstream.impl.storage.tenants;
+
+import lombok.Getter;
+
+public class TenantException extends Exception {
+
+    public enum Type {
+        NotFound,
+        AlreadyExists;
+    }
+
+    @Getter private final Type type;
+
+    public TenantException(String message, Type type) {
+        super(message);
+        this.type = type;
+    }
+}
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-api/pom.xml b/langstream-k8s-deployer/langstream-k8s-deployer-api/pom.xml
index 2920a0eba..f09b8bca5 100644
--- a/langstream-k8s-deployer/langstream-k8s-deployer-api/pom.xml
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-api/pom.xml
@@ -33,10 +33,6 @@
   </properties>
 
   <dependencies>
-    <dependency>
-      <groupId>io.fabric8</groupId>
-      <artifactId>kubernetes-client</artifactId>
-    </dependency>
     <dependency>
       <groupId>ai.langstream</groupId>
       <artifactId>langstream-api</artifactId>
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-core/pom.xml b/langstream-k8s-deployer/langstream-k8s-deployer-core/pom.xml
index 652424d2c..c433abbd9 100644
--- a/langstream-k8s-deployer/langstream-k8s-deployer-core/pom.xml
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-core/pom.xml
@@ -32,6 +32,10 @@
   </properties>
 
   <dependencies>
+    <dependency>
+      <groupId>io.fabric8</groupId>
+      <artifactId>kubernetes-client</artifactId>
+    </dependency>
     <dependency>
       <groupId>${project.groupId}</groupId>
       <artifactId>langstream-k8s-deployer-api</artifactId>
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-core/src/main/java/ai/langstream/deployer/k8s/PodTemplate.java b/langstream-k8s-deployer/langstream-k8s-deployer-core/src/main/java/ai/langstream/deployer/k8s/PodTemplate.java
index 2120344a3..3276cbc48 100644
--- a/langstream-k8s-deployer/langstream-k8s-deployer-core/src/main/java/ai/langstream/deployer/k8s/PodTemplate.java
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-core/src/main/java/ai/langstream/deployer/k8s/PodTemplate.java
@@ -24,7 +24,7 @@ public record PodTemplate(
         Map<String, String> nodeSelector,
         Map<String, String> annotations) {
 
-    static PodTemplate merge(PodTemplate primary, PodTemplate secondary) {
+    public static PodTemplate merge(PodTemplate primary, PodTemplate secondary) {
         return new PodTemplate(
                 merge(primary.tolerations(), secondary.tolerations()),
                 merge(primary.nodeSelector(), secondary.nodeSelector()),
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-core/src/main/java/ai/langstream/deployer/k8s/agents/AgentResourceUnitConfiguration.java b/langstream-k8s-deployer/langstream-k8s-deployer-core/src/main/java/ai/langstream/deployer/k8s/agents/AgentResourceUnitConfiguration.java
index ee11cc3c6..87e570c50 100644
--- a/langstream-k8s-deployer/langstream-k8s-deployer-core/src/main/java/ai/langstream/deployer/k8s/agents/AgentResourceUnitConfiguration.java
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-core/src/main/java/ai/langstream/deployer/k8s/agents/AgentResourceUnitConfiguration.java
@@ -31,5 +31,5 @@ public class AgentResourceUnitConfiguration {
     private int maxCpuMemUnits = 8;
     private int maxInstanceUnits = 8;
 
-    private int defaultMaxUnitsPerTenant = 0;
+    private int defaultMaxTotalResourceUnitsPerTenant = 0;
 }
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-operator/pom.xml b/langstream-k8s-deployer/langstream-k8s-deployer-operator/pom.xml
index 31ce1993d..eae081287 100644
--- a/langstream-k8s-deployer/langstream-k8s-deployer-operator/pom.xml
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-operator/pom.xml
@@ -41,6 +41,16 @@
       <artifactId>langstream-k8s-deployer-core</artifactId>
       <version>${project.version}</version>
     </dependency>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>langstream-api</artifactId>
+      <version>${project.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>langstream-k8s-storage</artifactId>
+      <version>${project.version}</version>
+    </dependency>
 
 
     <dependency>
@@ -92,6 +102,25 @@
       <artifactId>quarkus-junit5</artifactId>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>io.quarkus</groupId>
+      <artifactId>quarkus-test-kubernetes-client</artifactId>
+      <scope>test</scope>
+    </dependency>
+
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>langstream-k8s-common</artifactId>
+      <version>${project.version}</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>langstream-k8s-common</artifactId>
+      <version>${project.version}</version>
+      <classifier>tests</classifier>
+      <scope>test</scope>
+    </dependency>
   </dependencies>
 
   <build>
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/DeployerConfiguration.java b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/DeployerConfiguration.java
index b46fe67ee..e8043d0c2 100644
--- a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/DeployerConfiguration.java
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/DeployerConfiguration.java
@@ -22,6 +22,9 @@
 @ConfigMapping(prefix = "deployer")
 public interface DeployerConfiguration {
 
+    @WithDefault("{}")
+    String globalStorage();
+
     // workaround: quarkus doesn't support dynamic maps
     @WithDefault("{}")
     String clusterRuntime();
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/GlobalStorageConfiguration.java b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/GlobalStorageConfiguration.java
new file mode 100644
index 000000000..b2d9bd412
--- /dev/null
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/GlobalStorageConfiguration.java
@@ -0,0 +1,31 @@
+/*
+ * Copyright DataStax, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package ai.langstream.deployer.k8s;
+
+import java.util.HashMap;
+import java.util.Map;
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+public class GlobalStorageConfiguration {
+
+    private String type;
+    private Map<String, Object> configuration = new HashMap<>();
+}
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/ResolvedDeployerConfiguration.java b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/ResolvedDeployerConfiguration.java
index d213f2ea3..e52db89c1 100644
--- a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/ResolvedDeployerConfiguration.java
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/ResolvedDeployerConfiguration.java
@@ -22,12 +22,12 @@
 import com.fasterxml.jackson.databind.SerializationFeature;
 import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
 import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator;
-import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Singleton;
 import java.util.Map;
 import lombok.Getter;
 import lombok.SneakyThrows;
 
-@ApplicationScoped
+@Singleton
 public class ResolvedDeployerConfiguration {
 
     private static final ObjectMapper yamlMapper =
@@ -62,6 +62,9 @@ public ResolvedDeployerConfiguration(DeployerConfiguration configuration) {
 
         this.runtimeImage = configuration.runtimeImage().orElse(null);
         this.runtimeImagePullPolicy = configuration.runtimeImagePullPolicy().orElse(null);
+        this.globalStorageConfiguration =
+                yamlMapper.readValue(
+                        configuration.globalStorage(), GlobalStorageConfiguration.class);
     }
 
     @Getter private Map<String, Object> clusterRuntime;
@@ -75,4 +78,6 @@ public ResolvedDeployerConfiguration(DeployerConfiguration configuration) {
     @Getter private String runtimeImage;
 
     @Getter private String runtimeImagePullPolicy;
+
+    @Getter private GlobalStorageConfiguration globalStorageConfiguration;
 }
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/TenantLimitsChecker.java b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/TenantLimitsChecker.java
index fbc70902c..37b46ced4 100644
--- a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/TenantLimitsChecker.java
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/TenantLimitsChecker.java
@@ -15,6 +15,7 @@
  */
 package ai.langstream.deployer.k8s;
 
+import ai.langstream.api.webservice.tenant.TenantConfiguration;
 import ai.langstream.deployer.k8s.api.crds.apps.ApplicationCustomResource;
 import ai.langstream.deployer.k8s.limits.ApplicationResourceLimitsChecker;
 import ai.langstream.deployer.k8s.util.KeyedLockHandler;
@@ -27,27 +28,42 @@
 public class TenantLimitsChecker {
 
     private final ApplicationResourceLimitsChecker resourceLimitsEnforcer;
+    private final LimitsSupplier limitsSupplier;
 
     public TenantLimitsChecker(
             ResolvedDeployerConfiguration resolvedDeployerConfiguration,
-            KubernetesClient kubernetesClient) {
+            KubernetesClient kubernetesClient,
+            TenantsConfigurationReader tenantsConfigurationReader) {
+        this.limitsSupplier =
+                new LimitsSupplier(
+                        resolvedDeployerConfiguration
+                                .getAgentResources()
+                                .getDefaultMaxTotalResourceUnitsPerTenant(),
+                        tenantsConfigurationReader);
         this.resourceLimitsEnforcer =
                 new ApplicationResourceLimitsChecker(
-                        kubernetesClient,
-                        new KeyedLockHandler(),
-                        new LimitsSupplier(
-                                resolvedDeployerConfiguration
-                                        .getAgentResources()
-                                        .getDefaultMaxUnitsPerTenant()));
+                        kubernetesClient, new KeyedLockHandler(), limitsSupplier);
+    }
+
+    LimitsSupplier getLimitsSupplier() {
+        return limitsSupplier;
     }
 
     @AllArgsConstructor
     static class LimitsSupplier implements Function<String, Integer> {
         private final int defaultLimit;
+        private final TenantsConfigurationReader tenantsConfigurationReader;
 
         @Override
         public Integer apply(String s) {
-            return defaultLimit;
+            final TenantConfiguration config = tenantsConfigurationReader.getTenantConfiguration(s);
+            if (config == null) {
+                return defaultLimit;
+            }
+            if (config.getMaxTotalResourceUnits() <= 0) {
+                return defaultLimit;
+            }
+            return config.getMaxTotalResourceUnits();
         }
     }
 
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/TenantsConfigurationReader.java b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/TenantsConfigurationReader.java
new file mode 100644
index 000000000..e9a87070a
--- /dev/null
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/main/java/ai/langstream/deployer/k8s/TenantsConfigurationReader.java
@@ -0,0 +1,63 @@
+/*
+ * Copyright DataStax, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package ai.langstream.deployer.k8s;
+
+import ai.langstream.api.storage.GlobalMetadataStore;
+import ai.langstream.api.storage.GlobalMetadataStoreRegistry;
+import ai.langstream.api.webservice.tenant.TenantConfiguration;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import jakarta.enterprise.context.ApplicationScoped;
+import lombok.SneakyThrows;
+import lombok.extern.jbosslog.JBossLog;
+
+@ApplicationScoped
+@JBossLog
+public class TenantsConfigurationReader {
+    private static final ObjectMapper mapper =
+            new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+    private final GlobalMetadataStore globalMetadataStore;
+
+    public TenantsConfigurationReader(ResolvedDeployerConfiguration resolvedDeployerConfiguration) {
+        final GlobalStorageConfiguration configuration =
+                resolvedDeployerConfiguration.getGlobalStorageConfiguration();
+        if (configuration == null || configuration.getType() == null) {
+            log.warnf("No global storage configuration found. Tenants metadata won't be used.");
+            this.globalMetadataStore = null;
+        } else {
+            this.globalMetadataStore =
+                    GlobalMetadataStoreRegistry.loadStore(
+                            configuration.getType(), configuration.getConfiguration());
+        }
+    }
+
+    public TenantConfiguration getTenantConfiguration(String tenant) {
+        if (globalMetadataStore == null) {
+            return null;
+        }
+        final String config =
+                globalMetadataStore.get(GlobalMetadataStore.TENANT_KEY_PREFIX + tenant);
+        if (config == null) {
+            return null;
+        }
+        return parseTenantConfiguration(config);
+    }
+
+    @SneakyThrows
+    private TenantConfiguration parseTenantConfiguration(String res) {
+        return mapper.readValue(res, TenantConfiguration.class);
+    }
+}
diff --git a/langstream-k8s-deployer/pom.xml b/langstream-k8s-deployer/pom.xml
index 72bc2c36d..0f960be26 100644
--- a/langstream-k8s-deployer/pom.xml
+++ b/langstream-k8s-deployer/pom.xml
@@ -37,6 +37,8 @@
     <maven.compiler.source>17</maven.compiler.source>
     <maven.compiler.target>17</maven.compiler.target>
     <quarkus-operator-sdk-bom.version>6.2.1</quarkus-operator-sdk-bom.version>
+    <!--    override global okhttp3 v4 with v3, used by the k8s client -->
+    <okhttp.version>3.14.9</okhttp.version>
     <jsonassert.version>1.5.1</jsonassert.version>
   </properties>
 
@@ -54,6 +56,21 @@
         <artifactId>jsonassert</artifactId>
         <version>${jsonassert.version}</version>
       </dependency>
+      <dependency>
+        <groupId>com.squareup.okhttp3</groupId>
+        <artifactId>okhttp</artifactId>
+        <version>${okhttp.version}</version>
+      </dependency>
+      <dependency>
+        <groupId>com.squareup.okhttp3</groupId>
+        <artifactId>logging-interceptor</artifactId>
+        <version>${okhttp.version}</version>
+      </dependency>
+      <dependency>
+        <groupId>com.squareup.okhttp3</groupId>
+        <artifactId>mockwebserver</artifactId>
+        <version>${okhttp.version}</version>
+      </dependency>
     </dependencies>
   </dependencyManagement>
 
diff --git a/langstream-k8s-storage/src/main/java/ai/langstream/impl/storage/k8s/apps/KubernetesApplicationStore.java b/langstream-k8s-storage/src/main/java/ai/langstream/impl/storage/k8s/apps/KubernetesApplicationStore.java
index 29245ef0d..368ded083 100644
--- a/langstream-k8s-storage/src/main/java/ai/langstream/impl/storage/k8s/apps/KubernetesApplicationStore.java
+++ b/langstream-k8s-storage/src/main/java/ai/langstream/impl/storage/k8s/apps/KubernetesApplicationStore.java
@@ -52,7 +52,6 @@
 import java.util.stream.Collectors;
 import lombok.SneakyThrows;
 import lombok.extern.slf4j.Slf4j;
-import org.jetbrains.annotations.Nullable;
 
 @Slf4j
 public class KubernetesApplicationStore implements ApplicationStore {
@@ -89,6 +88,9 @@ public void onTenantCreated(String tenant) {
         new TenantResources(properties, client, tenant, namespace).ensureTenantResources();
     }
 
+    @Override
+    public void onTenantUpdated(String tenant) {}
+
     @Override
     public void onTenantDeleted(String tenant) {
         final String namespace = tenantToNamespace(tenant);
@@ -203,7 +205,6 @@ public Secrets getSecrets(String tenant, String applicationId) {
         return null;
     }
 
-    @Nullable
     private ApplicationCustomResource getApplicationCustomResource(
             String tenant, String applicationId) {
         final String namespace = tenantToNamespace(tenant);
diff --git a/langstream-runtime/langstream-runtime-impl/pom.xml b/langstream-runtime/langstream-runtime-impl/pom.xml
index 2244063d3..43d21eb96 100644
--- a/langstream-runtime/langstream-runtime-impl/pom.xml
+++ b/langstream-runtime/langstream-runtime-impl/pom.xml
@@ -495,7 +495,7 @@
                   <type>nar</type>
                   <classifier>nar</classifier>
                   <overWrite>false</overWrite>
-                  <outputDirectory>${build.directory}/agents</outputDirectory>
+                  <outputDirectory>${project.build.directory}/agents</outputDirectory>
                 </artifactItem>
 
                 <artifactItem>
@@ -505,7 +505,7 @@
                     <type>nar</type>
                     <classifier>nar</classifier>
                     <overWrite>false</overWrite>
-                    <outputDirectory>${build.directory}/agents</outputDirectory>
+                    <outputDirectory>${project.build.directory}/agents</outputDirectory>
                 </artifactItem>
 
                 <artifactItem>
@@ -515,7 +515,7 @@
                   <type>nar</type>
                   <classifier>nar</classifier>
                   <overWrite>false</overWrite>
-                  <outputDirectory>${build.directory}/agents</outputDirectory>
+                  <outputDirectory>${project.build.directory}/agents</outputDirectory>
                 </artifactItem>
 
                 <artifactItem>
@@ -525,7 +525,7 @@
                   <type>nar</type>
                   <classifier>nar</classifier>
                   <overWrite>false</overWrite>
-                  <outputDirectory>${build.directory}/agents</outputDirectory>
+                  <outputDirectory>${project.build.directory}/agents</outputDirectory>
                 </artifactItem>
 
                 <artifactItem>
@@ -535,7 +535,7 @@
                   <type>nar</type>
                   <classifier>nar</classifier>
                   <overWrite>false</overWrite>
-                  <outputDirectory>${build.directory}/agents</outputDirectory>
+                  <outputDirectory>${project.build.directory}/agents</outputDirectory>
                 </artifactItem>
 
               </artifactItems>
diff --git a/langstream-webservice/src/main/java/ai/langstream/webservice/LangStreamEventListener.java b/langstream-webservice/src/main/java/ai/langstream/webservice/LangStreamEventListener.java
index 949a08ba0..9c16a08cf 100644
--- a/langstream-webservice/src/main/java/ai/langstream/webservice/LangStreamEventListener.java
+++ b/langstream-webservice/src/main/java/ai/langstream/webservice/LangStreamEventListener.java
@@ -15,7 +15,7 @@
  */
 package ai.langstream.webservice;
 
-import ai.langstream.api.model.TenantConfiguration;
+import ai.langstream.api.webservice.tenant.TenantConfiguration;
 import ai.langstream.webservice.common.GlobalMetadataService;
 import ai.langstream.webservice.config.TenantProperties;
 import lombok.AllArgsConstructor;
diff --git a/langstream-webservice/src/main/java/ai/langstream/webservice/application/ApplicationService.java b/langstream-webservice/src/main/java/ai/langstream/webservice/application/ApplicationService.java
index dd76d824f..a00ca0e00 100644
--- a/langstream-webservice/src/main/java/ai/langstream/webservice/application/ApplicationService.java
+++ b/langstream-webservice/src/main/java/ai/langstream/webservice/application/ApplicationService.java
@@ -28,6 +28,7 @@
 import ai.langstream.api.runtime.PluginsRegistry;
 import ai.langstream.api.runtime.Topic;
 import ai.langstream.api.storage.ApplicationStore;
+import ai.langstream.api.webservice.tenant.TenantConfiguration;
 import ai.langstream.impl.common.DefaultAgentNode;
 import ai.langstream.impl.deploy.ApplicationDeployer;
 import ai.langstream.impl.parser.ModelBuilder;
@@ -92,8 +93,12 @@ public void deployApplication(
     }
 
     void checkResourceUsage(String tenant, String applicationId, ExecutionPlan executionPlan) {
-        final int max = tenantProperties.getDefaultMaxUnitsPerTenant();
-        if (max <= 0) {
+        final TenantConfiguration config = globalMetadataService.getTenant(tenant);
+        int maxTotalResourceUnits = config.getMaxTotalResourceUnits();
+        if (maxTotalResourceUnits <= 0) {
+            maxTotalResourceUnits = tenantProperties.getDefaultMaxTotalResourceUnits();
+        }
+        if (maxTotalResourceUnits <= 0) {
             return;
         }
 
@@ -104,7 +109,7 @@ void checkResourceUsage(String tenant, String applicationId, ExecutionPlan execu
                         .filter(e -> !e.getKey().equals(applicationId))
                         .collect(Collectors.summingInt(e -> e.getValue()));
         final int totalUsage = currentUsage + requestedUnits;
-        if (max < totalUsage) {
+        if (maxTotalResourceUnits < totalUsage) {
             throw new IllegalArgumentException(
                     "Not enough resources to deploy application " + applicationId);
         }
diff --git a/langstream-webservice/src/main/java/ai/langstream/webservice/common/GlobalMetadataService.java b/langstream-webservice/src/main/java/ai/langstream/webservice/common/GlobalMetadataService.java
index f92345f24..55ad198f3 100644
--- a/langstream-webservice/src/main/java/ai/langstream/webservice/common/GlobalMetadataService.java
+++ b/langstream-webservice/src/main/java/ai/langstream/webservice/common/GlobalMetadataService.java
@@ -15,12 +15,16 @@
  */
 package ai.langstream.webservice.common;
 
-import ai.langstream.api.model.TenantConfiguration;
 import ai.langstream.api.storage.ApplicationStore;
 import ai.langstream.api.storage.GlobalMetadataStore;
 import ai.langstream.api.storage.GlobalMetadataStoreRegistry;
+import ai.langstream.api.webservice.tenant.CreateTenantRequest;
+import ai.langstream.api.webservice.tenant.TenantConfiguration;
+import ai.langstream.api.webservice.tenant.UpdateTenantRequest;
 import ai.langstream.impl.storage.GlobalMetadataStoreManager;
+import ai.langstream.impl.storage.tenants.TenantException;
 import ai.langstream.webservice.config.StorageProperties;
+import ai.langstream.webservice.config.TenantProperties;
 import java.util.Map;
 import lombok.SneakyThrows;
 import org.springframework.stereotype.Service;
@@ -29,17 +33,49 @@
 public class GlobalMetadataService {
 
     private final GlobalMetadataStoreManager store;
+    private final TenantProperties tenantProperties;
 
     public GlobalMetadataService(
-            StorageProperties storageProperties, ApplicationStore applicationStore) {
+            StorageProperties storageProperties,
+            ApplicationStore applicationStore,
+            TenantProperties tenantProperties) {
         final GlobalMetadataStore globalMetadataStore =
                 GlobalMetadataStoreRegistry.loadStore(
                         storageProperties.getGlobal().getType(),
                         storageProperties.getGlobal().getConfiguration());
         store = new GlobalMetadataStoreManager(globalMetadataStore, applicationStore);
+        this.tenantProperties = tenantProperties;
+    }
+
+    public void createTenant(String tenant, CreateTenantRequest request) throws TenantException {
+        validateCreateTenantRequest(request);
+        store.createTenant(tenant, request);
+    }
+
+    private void validateCreateTenantRequest(CreateTenantRequest request) {
+        validateMaxTotalResourceUnits(request.getMaxTotalResourceUnits());
+    }
+
+    private void validateUpdateTenantRequest(UpdateTenantRequest request) {
+        validateMaxTotalResourceUnits(request.getMaxTotalResourceUnits());
+    }
+
+    private void validateMaxTotalResourceUnits(Integer maxTotalResourceUnits) {
+        if (tenantProperties.getMaxTotalResourceUnitsLimit() > 0
+                && maxTotalResourceUnits != null
+                && maxTotalResourceUnits > 0
+                && maxTotalResourceUnits > tenantProperties.getMaxTotalResourceUnitsLimit()) {
+            throw new IllegalArgumentException(
+                    "Max total resource units limit is "
+                            + tenantProperties.getMaxTotalResourceUnitsLimit());
+        }
+    }
+
+    public void updateTenant(String tenant, UpdateTenantRequest request) throws TenantException {
+        validateUpdateTenantRequest(request);
+        store.updateTenant(tenant, request);
     }
 
-    @SneakyThrows
     public void putTenant(String tenant, TenantConfiguration tenantConfiguration) {
         store.putTenant(tenant, tenantConfiguration);
     }
diff --git a/langstream-webservice/src/main/java/ai/langstream/webservice/common/ResourceErrorsHandler.java b/langstream-webservice/src/main/java/ai/langstream/webservice/common/ResourceErrorsHandler.java
index 655597ef9..5ff3bb679 100644
--- a/langstream-webservice/src/main/java/ai/langstream/webservice/common/ResourceErrorsHandler.java
+++ b/langstream-webservice/src/main/java/ai/langstream/webservice/common/ResourceErrorsHandler.java
@@ -38,7 +38,6 @@ ProblemDetail handleAll(Throwable exception) {
             log.error("Bad request", exception);
             return ProblemDetail.forStatusAndDetail(HttpStatus.BAD_REQUEST, exception.getMessage());
         }
-
         log.error("Internal error", exception);
         return ProblemDetail.forStatusAndDetail(
                 HttpStatus.INTERNAL_SERVER_ERROR, exception.getMessage());
diff --git a/langstream-webservice/src/main/java/ai/langstream/webservice/common/TenantResource.java b/langstream-webservice/src/main/java/ai/langstream/webservice/common/TenantResource.java
index 38e0d151d..36d9f63ff 100644
--- a/langstream-webservice/src/main/java/ai/langstream/webservice/common/TenantResource.java
+++ b/langstream-webservice/src/main/java/ai/langstream/webservice/common/TenantResource.java
@@ -15,7 +15,10 @@
  */
 package ai.langstream.webservice.common;
 
-import ai.langstream.api.model.TenantConfiguration;
+import ai.langstream.api.webservice.tenant.CreateTenantRequest;
+import ai.langstream.api.webservice.tenant.TenantConfiguration;
+import ai.langstream.api.webservice.tenant.UpdateTenantRequest;
+import ai.langstream.impl.storage.tenants.TenantException;
 import ai.langstream.webservice.application.ApplicationService;
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.tags.Tag;
@@ -23,10 +26,14 @@
 import java.util.Map;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.http.HttpStatus;
+import org.springframework.http.MediaType;
 import org.springframework.web.bind.annotation.DeleteMapping;
 import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PatchMapping;
 import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
 import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestBody;
 import org.springframework.web.bind.annotation.RequestMapping;
 import org.springframework.web.bind.annotation.RestController;
 import org.springframework.web.server.ResponseStatusException;
@@ -62,6 +69,42 @@ TenantConfiguration getTenant(@NotBlank @PathVariable("tenant") String tenant) {
         return configuration;
     }
 
+    @PostMapping(value = "/{tenant}", consumes = MediaType.APPLICATION_JSON_VALUE)
+    @Operation(summary = "Create a tenant")
+    void createTenant(
+            @NotBlank @PathVariable("tenant") String tenant,
+            @RequestBody CreateTenantRequest createTenantRequest)
+            throws TenantException {
+        try {
+            globalMetadataService.createTenant(tenant, createTenantRequest);
+        } catch (TenantException e) {
+            switch (e.getType()) {
+                case AlreadyExists:
+                    throw new ResponseStatusException(HttpStatus.CONFLICT, "tenant already exists");
+                default:
+                    throw e;
+            }
+        }
+    }
+
+    @PatchMapping(value = "/{tenant}", consumes = MediaType.APPLICATION_JSON_VALUE)
+    @Operation(summary = "Update a tenant")
+    void createTenant(
+            @NotBlank @PathVariable("tenant") String tenant,
+            @RequestBody UpdateTenantRequest updateTenantRequest)
+            throws TenantException {
+        try {
+            globalMetadataService.updateTenant(tenant, updateTenantRequest);
+        } catch (TenantException e) {
+            switch (e.getType()) {
+                case NotFound:
+                    throw new ResponseStatusException(HttpStatus.NOT_FOUND, "tenant not found");
+                default:
+                    throw e;
+            }
+        }
+    }
+
     @PutMapping(value = "/{tenant}")
     @Operation(summary = "Create or update a tenant")
     void putTenant(@NotBlank @PathVariable("tenant") String tenant) {
@@ -70,7 +113,7 @@ void putTenant(@NotBlank @PathVariable("tenant") String tenant) {
 
     @DeleteMapping("/{tenant}")
     @Operation(summary = "Delete tenant")
-    void deleteTenant(@NotBlank @PathVariable("tenant") String tenant) {
+    void deleteTenant(@NotBlank @PathVariable("tenant") String tenant) throws TenantException {
         applicationService
                 .getAllApplications(tenant)
                 .keySet()
diff --git a/langstream-webservice/src/main/java/ai/langstream/webservice/config/TenantProperties.java b/langstream-webservice/src/main/java/ai/langstream/webservice/config/TenantProperties.java
index ab8e907c5..c203335b3 100644
--- a/langstream-webservice/src/main/java/ai/langstream/webservice/config/TenantProperties.java
+++ b/langstream-webservice/src/main/java/ai/langstream/webservice/config/TenantProperties.java
@@ -36,5 +36,6 @@ public static class DefaultTenantProperties {
     }
 
     private DefaultTenantProperties defaultTenant;
-    private int defaultMaxUnitsPerTenant;
+    private int defaultMaxTotalResourceUnits;
+    private int maxTotalResourceUnitsLimit;
 }
diff --git a/langstream-webservice/src/main/java/ai/langstream/webservice/security/infrastructure/primary/LocalKubernetesJwksUriSigningKeyResolver.java b/langstream-webservice/src/main/java/ai/langstream/webservice/security/infrastructure/primary/LocalKubernetesJwksUriSigningKeyResolver.java
index 61684bf63..5b226a71f 100644
--- a/langstream-webservice/src/main/java/ai/langstream/webservice/security/infrastructure/primary/LocalKubernetesJwksUriSigningKeyResolver.java
+++ b/langstream-webservice/src/main/java/ai/langstream/webservice/security/infrastructure/primary/LocalKubernetesJwksUriSigningKeyResolver.java
@@ -27,7 +27,6 @@
 import java.util.concurrent.ConcurrentHashMap;
 import lombok.SneakyThrows;
 import lombok.extern.slf4j.Slf4j;
-import org.jetbrains.annotations.Nullable;
 
 @Slf4j
 public class LocalKubernetesJwksUriSigningKeyResolver {
@@ -108,7 +107,6 @@ private static String composeWellKnownEndpoint(String issuer) {
         return kubeOpenIDUrl;
     }
 
-    @Nullable
     private JwksUriSigningKeyResolver.JwksUri getJwksUri(String kubeOpenIDUrl) {
         try {
             final Map<String, ?> response =
diff --git a/pom.xml b/pom.xml
index 3d6e29437..0e26e9e9b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -76,6 +76,8 @@
     <json-schema-validator.version>1.0.72</json-schema-validator.version>
     <tomcat-embed-el.version>10.1.4</tomcat-embed-el.version>
     <commons-collections4.version>4.4</commons-collections4.version>
+<!--    minio requires okhttp3 v4-->
+    <okhttp.version>4.11.0</okhttp.version>
   </properties>
 
   <scm>
@@ -396,6 +398,21 @@
         <artifactId>jetty-servlet</artifactId>
         <version>${jetty.version}</version>
       </dependency>
+      <dependency>
+        <groupId>com.squareup.okhttp3</groupId>
+        <artifactId>okhttp</artifactId>
+        <version>${okhttp.version}</version>
+      </dependency>
+      <dependency>
+        <groupId>com.squareup.okhttp3</groupId>
+        <artifactId>logging-interceptor</artifactId>
+        <version>${okhttp.version}</version>
+      </dependency>
+      <dependency>
+        <groupId>com.squareup.okhttp3</groupId>
+        <artifactId>mockwebserver</artifactId>
+        <version>${okhttp.version}</version>
+      </dependency>
     </dependencies>
 
   </dependencyManagement>


Test patch:
diff --git a/langstream-cli/src/test/java/ai/langstream/cli/commands/applications/TenantsCmdTest.java b/langstream-cli/src/test/java/ai/langstream/cli/commands/applications/TenantsCmdTest.java
index 2e981ade0..a761c83b7 100644
--- a/langstream-cli/src/test/java/ai/langstream/cli/commands/applications/TenantsCmdTest.java
+++ b/langstream-cli/src/test/java/ai/langstream/cli/commands/applications/TenantsCmdTest.java
@@ -15,6 +15,8 @@
  */
 package ai.langstream.cli.commands.applications;
 
+import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
+
 import com.github.tomakehurst.wiremock.client.WireMock;
 import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.Test;
@@ -32,6 +34,44 @@ public void testPut() {
         Assertions.assertEquals("tenant newt created/updated", result.out());
     }
 
+    @Test
+    public void testCreate() {
+        wireMock.register(
+                WireMock.post("/api/tenants/%s".formatted("newt"))
+                        .withRequestBody(WireMock.equalToJson("{\"maxTotalResourceUnits\":null}"))
+                        .willReturn(WireMock.ok("")));
+        CommandResult result = executeCommand("tenants", "create", "newt");
+        Assertions.assertEquals(0, result.exitCode());
+        Assertions.assertEquals("", result.err());
+        Assertions.assertEquals("tenant newt created", result.out());
+    }
+
+    @Test
+    public void testCreatemaxTotalResourceUnits() {
+        wireMock.register(
+                WireMock.post("/api/tenants/%s".formatted("newt"))
+                        .withRequestBody(WireMock.equalToJson("{\"maxTotalResourceUnits\":10}"))
+                        .willReturn(WireMock.ok("")));
+        CommandResult result =
+                executeCommand("tenants", "create", "newt", "--max-total-resource-units", "10");
+        Assertions.assertEquals(0, result.exitCode());
+        Assertions.assertEquals("", result.err());
+        Assertions.assertEquals("tenant newt created", result.out());
+    }
+
+    @Test
+    public void testUpdatemaxTotalResourceUnits() {
+        wireMock.register(
+                WireMock.patch(urlEqualTo("/api/tenants/%s".formatted("newt")))
+                        .withRequestBody(WireMock.equalToJson("{\"maxTotalResourceUnits\":10}"))
+                        .willReturn(WireMock.ok("")));
+        CommandResult result =
+                executeCommand("tenants", "update", "newt", "--max-total-resource-units", "10");
+        Assertions.assertEquals(0, result.exitCode());
+        Assertions.assertEquals("", result.err());
+        Assertions.assertEquals("tenant newt updated", result.out());
+    }
+
     @Test
     public void testGet() {
         wireMock.register(
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/test/java/ai/langstream/deployer/k8s/TenantLimitsCheckerTest.java b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/test/java/ai/langstream/deployer/k8s/TenantLimitsCheckerTest.java
new file mode 100644
index 000000000..ebc7acbb7
--- /dev/null
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/test/java/ai/langstream/deployer/k8s/TenantLimitsCheckerTest.java
@@ -0,0 +1,86 @@
+/*
+ * Copyright DataStax, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package ai.langstream.deployer.k8s;
+
+import io.fabric8.kubernetes.api.model.ConfigMapBuilder;
+import io.fabric8.kubernetes.client.server.mock.KubernetesServer;
+import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.QuarkusTestProfile;
+import io.quarkus.test.junit.TestProfile;
+import io.quarkus.test.kubernetes.client.KubernetesTestServer;
+import io.quarkus.test.kubernetes.client.WithKubernetesTestServer;
+import jakarta.inject.Inject;
+import java.util.Map;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+@QuarkusTest
+@WithKubernetesTestServer
+@TestProfile(TenantLimitsCheckerTest.MyTestProfile.class)
+class TenantLimitsCheckerTest {
+
+    @KubernetesTestServer KubernetesServer mockServer;
+
+    public static class MyTestProfile implements QuarkusTestProfile {
+
+        @Override
+        public Map<String, String> getConfigOverrides() {
+            return Map.of(
+                    "deployer.agent-resources",
+                    "{\"defaultMaxTotalResourceUnitsPerTenant\":10}",
+                    "deployer.global-storage",
+                    "{\"type\":\"kubernetes\",\"configuration\":{\"namespace\":\"default\"}}");
+        }
+    }
+
+    @Inject TenantLimitsChecker checker;
+
+    @BeforeEach
+    public void before() {
+        mockServer
+                .getKubernetesMockServer()
+                .expect()
+                .get()
+                .withPath("/api/v1/namespaces/default/configmaps/langstream-t-my-tenant")
+                .andReturn(
+                        200,
+                        new ConfigMapBuilder()
+                                .withData(Map.of("value", "{\"maxTotalResourceUnits\":20}"))
+                                .build())
+                .always();
+
+        mockServer
+                .getKubernetesMockServer()
+                .expect()
+                .get()
+                .withPath("/api/v1/namespaces/default/configmaps/langstream-t-my-tenant3")
+                .andReturn(
+                        200,
+                        new ConfigMapBuilder()
+                                .withData(Map.of("value", "{\"maxTotalResourceUnits\":0}"))
+                                .build())
+                .always();
+    }
+
+    @Test
+    void testLimitsSupplier() {
+        final TenantLimitsChecker.LimitsSupplier supplier = checker.getLimitsSupplier();
+        Assertions.assertEquals(10, supplier.apply("my-tenant2"));
+        Assertions.assertEquals(10, supplier.apply("my-tenant3"));
+        Assertions.assertEquals(20, supplier.apply("my-tenant"));
+    }
+}
diff --git a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/test/java/ai/langstream/deployer/k8s/controllers/AppControllerIT.java b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/test/java/ai/langstream/deployer/k8s/controllers/AppControllerIT.java
index 2869170bf..610a4e47b 100644
--- a/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/test/java/ai/langstream/deployer/k8s/controllers/AppControllerIT.java
+++ b/langstream-k8s-deployer/langstream-k8s-deployer-operator/src/test/java/ai/langstream/deployer/k8s/controllers/AppControllerIT.java
@@ -43,7 +43,8 @@ public class AppControllerIT {
     static final OperatorExtension deployment =
             new OperatorExtension(
                     Map.of(
-                            "DEPLOYER_AGENT_RESOURCES", "{defaultMaxUnitsPerTenant: 3}",
+                            "DEPLOYER_AGENT_RESOURCES",
+                                    "{defaultMaxTotalResourceUnitsPerTenant: 3}",
                             "DEPLOYER_RUNTIME_IMAGE", "busybox",
                             "DEPLOYER_RUNTIME_IMAGE_PULL_POLICY", "IfNotPresent"));
 
diff --git a/langstream-webservice/src/test/java/ai/langstream/webservice/WebAppTestConfig.java b/langstream-webservice/src/test/java/ai/langstream/webservice/WebAppTestConfig.java
index d04ca984b..ac4585be0 100644
--- a/langstream-webservice/src/test/java/ai/langstream/webservice/WebAppTestConfig.java
+++ b/langstream-webservice/src/test/java/ai/langstream/webservice/WebAppTestConfig.java
@@ -27,10 +27,8 @@
 @TestConfiguration
 public class WebAppTestConfig {
 
-    @Bean
-    @Primary
     @SneakyThrows
-    public StorageProperties storageProperties() {
+    public static StorageProperties buildStorageProperties() {
         return new StorageProperties(
                 new StorageProperties.AppsStoreProperties(
                         "kubernetes", Map.of("namespaceprefix", "langstream-")),
@@ -43,4 +41,11 @@ public StorageProperties storageProperties() {
                                         .getAbsolutePath())),
                 new StorageProperties.CodeStorageProperties());
     }
+
+    @Bean
+    @Primary
+    @SneakyThrows
+    public StorageProperties storageProperties() {
+        return buildStorageProperties();
+    }
 }
diff --git a/langstream-webservice/src/test/java/ai/langstream/webservice/application/ApplicationServiceResourceLimitTest.java b/langstream-webservice/src/test/java/ai/langstream/webservice/application/ApplicationServiceResourceLimitTest.java
index 99d18c6bc..a3e2942d9 100644
--- a/langstream-webservice/src/test/java/ai/langstream/webservice/application/ApplicationServiceResourceLimitTest.java
+++ b/langstream-webservice/src/test/java/ai/langstream/webservice/application/ApplicationServiceResourceLimitTest.java
@@ -21,9 +21,14 @@
 import ai.langstream.api.model.StoredApplication;
 import ai.langstream.api.runtime.ExecutionPlan;
 import ai.langstream.api.storage.ApplicationStore;
+import ai.langstream.api.webservice.tenant.TenantConfiguration;
+import ai.langstream.api.webservice.tenant.UpdateTenantRequest;
 import ai.langstream.impl.k8s.tests.KubeTestServer;
 import ai.langstream.impl.parser.ModelBuilder;
+import ai.langstream.webservice.WebAppTestConfig;
+import ai.langstream.webservice.common.GlobalMetadataService;
 import ai.langstream.webservice.config.ApplicationDeployProperties;
+import ai.langstream.webservice.config.StorageProperties;
 import ai.langstream.webservice.config.TenantProperties;
 import java.util.List;
 import java.util.Map;
@@ -39,9 +44,15 @@
 class ApplicationServiceResourceLimitTest {
 
     @RegisterExtension static final KubeTestServer k3s = new KubeTestServer();
+    protected static final StorageProperties STORAGE_PROPERTIES =
+            WebAppTestConfig.buildStorageProperties();
 
     @Test
     void test() {
+        new GlobalMetadataService(STORAGE_PROPERTIES, new TestApplicationStore(Map.of()), null)
+                .putTenant("tenant", TenantConfiguration.builder().build());
+
+        putTenantLimit(0);
         validate("app1", filesWithTwoAgents(100, 100), Map.of(), 0, true);
         validate("app1", filesWithTwoAgents(1, 2), Map.of(), 1, false);
         validate("app1", filesWithTwoAgents(1, 2), Map.of(), 2, true);
@@ -56,6 +67,12 @@ void test() {
 
         validate("app1", filesWithTwoAgents(1, 2), Map.of("app1", 2, "app2", 2), 4, true);
         validate("app3", filesWithTwoAgents(1, 2), Map.of("app1", 2, "app2", 2), 4, false);
+        putTenantLimit(6);
+        validate("app3", filesWithTwoAgents(1, 2), Map.of("app1", 2, "app2", 2), 4, true);
+        putTenantLimit(4);
+        validate("app3", filesWithTwoAgents(1, 2), Map.of("app1", 2, "app2", 2), 4, false);
+        putTenantLimit(0);
+        validate("app3", filesWithTwoAgents(1, 2), Map.of("app1", 2, "app2", 2), 4, false);
     }
 
     private Map<String, String> filesWithTwoAgents(int size, int parallelism) {
@@ -95,12 +112,12 @@ private static void validate(
                 ModelBuilder.buildApplicationInstance(
                                 files,
                                 """
-                        instance:
-                          streamingCluster:
-                            type: "noop"
-                          computeCluster:
-                            type: "kubernetes"
-                        """,
+                                        instance:
+                                          streamingCluster:
+                                            type: "noop"
+                                          computeCluster:
+                                            type: "kubernetes"
+                                        """,
                                 null)
                         .getApplication();
 
@@ -134,14 +151,26 @@ private static void validate(
         }
     }
 
+    @SneakyThrows
+    private void putTenantLimit(int limit) {
+        new GlobalMetadataService(
+                        STORAGE_PROPERTIES,
+                        new TestApplicationStore(Map.of()),
+                        new TenantProperties())
+                .updateTenant(
+                        "tenant",
+                        UpdateTenantRequest.builder().maxTotalResourceUnits(limit).build());
+    }
+
     @NotNull
     private static ApplicationService getApplicationService(
             Map<String, Integer> currentUsage, int max) {
         final TenantProperties props = new TenantProperties();
-        props.setDefaultMaxUnitsPerTenant(max);
+        props.setDefaultMaxTotalResourceUnits(max);
+        final TestApplicationStore applicationStore = new TestApplicationStore(currentUsage);
         return new ApplicationService(
-                null,
-                new TestApplicationStore(currentUsage),
+                new GlobalMetadataService(STORAGE_PROPERTIES, applicationStore, props),
+                applicationStore,
                 new ApplicationDeployProperties(
                         new ApplicationDeployProperties.GatewayProperties(false)),
                 props);
@@ -152,14 +181,13 @@ private static class TestApplicationStore implements ApplicationStore {
         private final Map<String, Integer> currentUsage;
 
         @Override
-        public void onTenantCreated(String tenant) {
-            throw new UnsupportedOperationException();
-        }
+        public void onTenantCreated(String tenant) {}
 
         @Override
-        public void onTenantDeleted(String tenant) {
-            throw new UnsupportedOperationException();
-        }
+        public void onTenantDeleted(String tenant) {}
+
+        @Override
+        public void onTenantUpdated(String tenant) {}
 
         @Override
         public void put(
diff --git a/langstream-webservice/src/test/java/ai/langstream/webservice/application/TenantResourceTest.java b/langstream-webservice/src/test/java/ai/langstream/webservice/application/TenantResourceTest.java
index 5506624b8..d460bfa05 100644
--- a/langstream-webservice/src/test/java/ai/langstream/webservice/application/TenantResourceTest.java
+++ b/langstream-webservice/src/test/java/ai/langstream/webservice/application/TenantResourceTest.java
@@ -17,6 +17,8 @@
 
 import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.delete;
 import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
+import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.patch;
+import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
 import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.put;
 import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
 import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
@@ -33,7 +35,7 @@
 import org.springframework.test.annotation.DirtiesContext;
 import org.springframework.test.web.servlet.MockMvc;
 
-@SpringBootTest
+@SpringBootTest(properties = {"application.tenants.max-total-resource-units-limit=8"})
 @AutoConfigureMockMvc
 @Slf4j
 @Import(WebAppTestConfig.class)
@@ -49,13 +51,70 @@ void testTenantCrud() throws Exception {
         mockMvc.perform(put("/api/tenants/my-tenant")).andExpect(status().isOk());
         mockMvc.perform(put("/api/tenants/test")).andExpect(status().isOk());
 
+        mockMvc.perform(
+                        patch("/api/tenants/test")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":1}"))
+                .andExpect(status().isOk());
+        mockMvc.perform(
+                        patch("/api/tenants/test")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":0}"))
+                .andExpect(status().isOk());
+
+        mockMvc.perform(
+                        patch("/api/tenants/test")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":8}"))
+                .andExpect(status().isOk());
+        mockMvc.perform(
+                        patch("/api/tenants/test")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":-111}"))
+                .andExpect(status().isBadRequest());
+
+        mockMvc.perform(
+                        patch("/api/tenants/test")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":9}"))
+                .andExpect(status().isBadRequest());
+
+        mockMvc.perform(
+                        patch("/api/tenants/test2")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":0}"))
+                .andExpect(status().isNotFound());
+
+        mockMvc.perform(
+                        post("/api/tenants/test")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":0}"))
+                .andExpect(status().isConflict());
+
+        mockMvc.perform(
+                        post("/api/tenants/test2")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":1}"))
+                .andExpect(status().isOk());
+        mockMvc.perform(
+                        post("/api/tenants/test3")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":0}"))
+                .andExpect(status().isOk());
+        mockMvc.perform(
+                        post("/api/tenants/test4")
+                                .contentType("application/json")
+                                .content("{\"maxTotalResourceUnits\":-111}"))
+                .andExpect(status().isBadRequest());
+
         mockMvc.perform(get("/api/tenants/test"))
                 .andExpect(status().isOk())
                 .andExpect(jsonPath("$.name").value("test"));
 
         mockMvc.perform(get("/api/tenants"))
                 .andExpect(status().isOk())
-                .andExpect(jsonPath("$.test.name").value("test"));
+                .andExpect(jsonPath("$.test.name").value("test"))
+                .andExpect(jsonPath("$.test.maxTotalResourceUnits").value(8));
 
         mockMvc.perform(delete("/api/tenants/test")).andExpect(status().isOk());