Chapter 1. Installing 3scale API Management on OpenShift This section walks you through steps to deploy Red Hat 3scale API Management 2.15 on OpenShift. The 3scale solution for on-premises deployment is composed of: Two application programming interface (API) gateways: embedded APIcast. One 3scale Admin Portal and Developer Portal with persistent storage. Note The 3scale Istio Adapter is available as an optional adapter that allows labeling a service running within the Red Hat OpenShift Service Mesh, and integrate that service with 3scale. Refer to 3scale adapter documentation for more information. Prerequisites You must configure 3scale servers for UTC (Coordinated Universal Time). To install 3scale on OpenShift, perform the steps outlined in the following sections: System requirements for installing 3scale API Management on OpenShift Deploying 3scale API Management using the operator External databases for 3scale API Management using the operator Deployment configuration options for 3scale API Management on OpenShift using the operator Installing 3scale API Management with the operator using Oracle as the system database Troubleshooting common 3scale API Management installation issues 1.1. System requirements for installing 3scale API Management on OpenShift This section lists the system requirements for installing Red Hat 3scale API Management on OpenShift. 1.1.1. Environment requirements 3scale requires an environment specified in supported configurations . Note The requirements for persistent volumes vary between different deployment types. When deploying with external databases, persistent volumes are not necessary. For some deployment types, an Amazon S3 bucket can serve as a substitute for persistent volumes. If you use local file system storage, consider the specific deployment type and its associated requirements for persistent volumes. Persistent volumes 4 RWO (ReadWriteOnce) persistent volumes for Redis, MySQL, and System-searchd persistence. 1 RWX (ReadWriteMany) persistent volume for Developer Portal content and System-app Assets. Configure the RWX persistent volume to be group writable. For a list of persistent volume types that support the required access modes, see the OpenShift documentation . Note Network File System (NFS) is supported on 3scale for the RWX volume only. For IBM Power (ppc64le) and IBM Z (s390x), provision local storage using the following: Storage NFS If you are using an Amazon Simple Storage Service (Amazon S3) bucket for content management system (CMS) storage: Persistent volumes 3 RWO (ReadWriteOnce) persistent volumes for Redis and MySQL persistence. Storage 1 Amazon S3 bucket NFS 1.1.2. Hardware requirements Hardware requirements depend on your usage needs. Red Hat recommends that you test and configure your environment to meet your specific requirements. The following are the recommendations when configuring your environment for 3scale on OpenShift: Compute optimized nodes for deployments on cloud environments (AWS c4.2xlarge or Azure Standard_F8). Very large installations may require a separate node (AWS M4 series or Azure Av2 series) for Redis if memory requirements exceed your current node's available RAM. Note that this is true only when the deployment is with embedded Redis. Separate nodes between routing and compute tasks. Dedicated computing nodes for 3scale specific tasks. Additional resources Understanding persistent storage 1.2. Installing the 3scale API Management operator on OpenShift Note 3scale supports the last two general availability (GA) releases of OpenShift Container Platform (OCP). For more information, see the Red Hat 3scale API Management Supported Configurations page. This documentation shows you how to: Create a new project. Deploy a Red Hat 3scale API Management instance. Deploy the custom resources once the operator has been deployed. Prerequisites Access to a supported version of an OpenShift Container Platform 4 cluster using an account with administrator privileges. For more information about supported configurations, see the Red Hat 3scale API Management Supported Configurations page. Warning Deploy the 3scale operator and custom resource definitions (CRDs) in a separate newly created, empty project . If you deploy them in an existing project containing infrastructure, it could alter or delete existing elements. To install the 3scale operator on OpenShift, perform the steps outlined in the following sections: Creating a new OpenShift project Installing and configuring the 3scale API Management operator using the OLM 1.2.1. Creating a new OpenShift project This procedure explains how to create a new OpenShift project named 3scale-project . Replace this project name with your own. Procedure To create a new OpenShift project: Indicate a valid name using alphanumeric characters and dashes. As an example, run the command below to create 3scale-project : This creates the new OpenShift project where the operator, the APIManager custom resource (CR), and the Capabilities custom resources will be installed. The operator manages the custom resources through OLM in that project. 1.2.2. Installing and configuring the 3scale API Management operator using the OLM Use Operator Lifecycle Manager (OLM) to install the 3scale operator on an OpenShift Container Platform (OCP) 4.12 (or above) cluster through the OperatorHub in the OCP console. You can install the 3scale operator using the following installation modes: Cluster-wide in which the operator is available in all namespaces on the cluster. A specific namespace on the cluster Note If you are using the OpenShift Container Platform on a restricted network or a disconnected cluster, the Operator Lifecycle Manager can no longer use the OperatorHub. Follow the instructions for setting up and using the OLM in the guide titled Using Operator Lifecycle Manager on restricted networks . Prerequisites You must install and deploy the 3scale operator in the project that you defined in Creating a new OpenShift project . Procedure In the OpenShift Container Platform console, log in using an account with administrator privileges. Click Operators > OperatorHub In the Filter by keyword box, type 3scale operator to find Red Hat Integration - 3scale . Click Red Hat Integration - 3scale . Information about the operator is displayed. Read the information about the operator and click Install . The Install Operator page opens. On the Install Operator page, select the desired channel to update in the Update channel section. In the Installation mode section, select where to install the operator. All namespaces on the cluster (default) - The operator will be available in all namespaces on the cluster. A specific namespace on the cluster - The operator will only be available in the specific single namespace on the cluster that you have selected. Click Install . After the installation is complete, the system displays a confirmation message indicating that the operator is ready for use. Verify that the 3scale operator ClusterServiceVersion (CSV) is correctly installed. Also check if it reports that the installation of the operator has been successful: Click Operators > Installed Operators . Click on the Red Hat Integration - 3scale operator. In the Details tab, scroll down to the Conditions section, where the Succeeded condition should read InstallSucceeded under the Reason column. Besides the indicated procedure, create a list of the allowed domains you intend to use in the 3scale Developer Portal while using OCP on restricted networks. Consider the following examples: Any link you intend to add to the Developer Portal. Single sign-on (SSO) integrations through third party SSO providers such as GitHub. Billing. Webhooks that trigger an external URL. 1.2.2.1. Restrictions in disconnected environments The following list outlines current restrictions in a disconnected environment for 3scale 2.15: The GitHub login to the Developer Portal is not available. Support links are not operational. Links to external documentation are not operational. The validator for the OpenAPI Specification (OAS) in the Developer Portal is not operational, affecting links to external services. In the product Overview page in ActiveDocs , links to OAS are not operational. It is also necessary to check the option Skip swagger validations when you create a new ActiveDocs specification. Additional resources OpenShift Container Platform documentation Using Operator Lifecycle Manager on restricted networks Mirroring images for a disconnected installation Red Hat 3scale API Management Supported Configurations 1.2.3. Upgrading the 3scale API Management operator using the OLM To upgrade the 3scale operator from a single namespace to a cluster-wide installation in all namespaces on an operator-based deployment, you must remove the 3scale operator from the namespace and then reinstall the operator on the cluster. Cluster administrators can delete installed operators from a selected namespace by using the web console. Uninstalling the operator does not uninstall an existing 3scale instance. After the 3scale operator is uninstalled from the namespace, you can use OLM to install the operator in the cluster-wide mode. Prerequisites 3scale administrator permissions or an OpenShift role that has delete permissions for the namespace. Procedure In the OpenShift Container Platform console, log in using an account with administrator privileges. Click Operators > OperatorHub . The installed Operators page is displayed. Enter 3scale into the Filter by name to find the operator and click on it. On the Operator Details page, select Uninstall Operator from the Actions drop-down menu to remove it from a specific namespace. An Uninstall Operator? dialog box is displayed, reminding you that: Select Uninstall . This operator stops running and no longer receives updates. In the OpenShift Container Platform console click Operators > OperatorHub . In the Filter by keyword box, type 3scale operator to find Red Hat Integration - 3scale . Click Red Hat Integration - 3scale . Information about the operator is displayed. Click Install . The Install Operator page opens. On the Install Operator page, select the desired channel to update in the Update channel section. In the Installation mode section, select All namespaces on the cluster (default) . The operator will be available in all namespaces on the cluster. Click Subscribe . The 3scale operator details page is displayed and you can see the Subscription Overview . Confirm that the subscription Upgrade Status is displayed as Up to date . Verify that the 3scale operator ClusterServiceVersion (CSV) is displayed. Additional Resources Installing the 3scale API Management operator on OpenShift 1.2.3.1. Configuring automated application of micro releases Warning If you are using an external Oracle database, set the 3scale update strategy to Manual . With an external Oracle database, the database and the .spec.system.image are updated manually. The Automatic setting would not update the .spec.system.image . See the link:h:Link3scaleMigrating3scale: Migrating 3scale guide to update an operator-based installation with an external Oracle database. To get automatic updates, the 3scale operator must have its approval strategy set to Automatic . This allows it to apply micro release updates automatically. The following describes the differences between Automatic and Manual settings, and outlines the steps in a procedure to change from one to the other. Automatic and manual: During installation, the Automatic setting is the selected option by default. Installation of new updates occur as they become available. You can change this during the install or at any time afterwards. If you select the Manual option during installation or at any time afterwards, you will receive updates when they are available. , you must approve the Install Plan and apply it yourself. Procedure Click Operators > Installed Operators . Click Red Hat Integration - 3scale from the list of Installed Operators . Click the Subscription tab. Under the Subscription Details heading, you will see the subheading Approval . Click the link below Approval . The link is set to Automatic by default. A modal with the heading Change Update Approval Strategy will pop up. Choose the option of your preference: Automatic (default) or Manual , and then click Save . Additional resources Installing Operators in your namespace 1.3. Installing the APIcast operator on OpenShift This guide provides steps for installing the APIcast operator through the OpenShift Container Platform (OCP) console. Prerequisites OCP 4.x or later with administrator privileges. Procedure Create new project operator-test in Projects > Create Project . Click Operators > OperatorHub . In the Filter by keyword box, type apicast operator to find Red Hat Integration - 3scale APIcast gateway . Click Red Hat Integration - 3scale APIcast gateway . Information about the APIcast operator is displayed. Click Install . The Create Operator Subscription page opens. Click Install to accept all of the default selections on the Create Operator Subscription page. Note You can select different operator versions and installation modes, such as cluster-wide or namespace-specific options. There can only be one cluster-wide installation per cluster. The subscription Upgrade Status is shown as Up to date . Click Operators > Installed Operators to verify that the APIcast operator ClusterServiceVersion (CSV) status displays to InstallSucceeded in the operator-test project. 1.4. Deploying 3scale API Management using the operator This section takes you through installing and deploying the 3scale solution via the 3scale operator, using the APIManager custom resource (CR). Note Wildcard routes have been removed since 3scale 2.6. This functionality is handled by Zync in the background. When API providers are created, updated, or deleted, routes automatically reflect those changes. Prerequisites To make sure you receive automatic updates of micro releases for 3scale, you must have enabled the automatic approval functionality in the 3scale operator. Automatic is the default approval setting. To change this at any time based on your specific needs, use the steps for Configuring automated application of micro releases . Deploying 3scale API Management using the operator first requires that you follow the steps in Installing the 3scale API Management operator on OpenShift OpenShift Container Platform 4 A user account with administrator privileges in the OpenShift cluster. For more information about supported configurations, see the Red Hat 3scale API Management Supported Configurations page. Follow these procedures to deploy 3scale using the operator: Deploying the APIManager custom resource Getting the Admin Portal URL Getting the APIManager Admin Portal and Master Admin Portal credentials External databases for 3scale API Management using the operator 1.4.1. Deploying the APIManager custom resource Note If you decide to use Amazon Simple Storage Service (Amazon S3), see Amazon Simple Storage Service 3scale API Management fileStorage installation . The operator watches for APIManager CRs and deploys your required 3scale solution as specified in the APIManager CR. Procedure Click Operators > Installed Operators . From the list of Installed Operators , click Red Hat Integration - 3scale . Click the API Manager tab. Click Create APIManager . Clear the sample content and add the following YAML definitions to the editor, then click Create . Before 3scale 2.8, you could configure the automatic addition of replicas by setting the highAvailability field to true . From 3scale 2.8, the addition of replicas is controlled through the replicas field in the APIManager CR as shown in the following example. Note The value of the wildcardDomain parameter must be a valid domain name that resolves to the address of your OpenShift Container Platform (OCP) router. For example, apps.mycluster.example.com . APIManager CR with minimum requirements: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager-sample spec: wildcardDomain: example.com APIManager CR with replicas configured: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager-sample spec: system: appSpec: replicas: 1 sidekiqSpec: replicas: 1 zync: appSpec: replicas: 1 queSpec: replicas: 1 backend: cronSpec: replicas: 1 listenerSpec: replicas: 1 workerSpec: replicas: 1 apicast: productionSpec: replicas: 1 stagingSpec: replicas: 1 wildcardDomain: example.com 1.4.2. Getting the Admin Portal URL When you deploy 3scale using the operator, a default tenant is created with a fixed URL: 3scale-admin.USD{wildcardDomain} . The wildcardDomain is the <wildCardDomain> parameter you provided during the installation. Open this unique URL in a browser using the this command: Optionally, you can create new tenants on the MASTER portal URL : master.USD{wildcardDomain} . 1.4.3. Getting the APIManager Admin Portal and Master Admin Portal credentials Note Replace <namespace> with the name of the namespace where the APIManager resource was created. To log in to either the 3scale Admin Portal or Master Admin Portal after the operator-based deployment, you need the credentials for each separate portal. To get these credentials: Run the following commands to get the Admin Portal credentials: USD oc get secret system-seed -n <namespace> -o json | jq -r .data.ADMIN_USER | base64 -d USD oc get secret system-seed -n <namespace> -o json | jq -r .data.ADMIN_PASSWORD | base64 -d Log in as the Admin Portal administrator to verify these credentials are working. Run the following commands to get the Master Admin Portal credentials: USD oc get secret system-seed -n <namespace> -o json | jq -r .data.MASTER_USER | base64 -d USD oc get secret system-seed -n <namespace> -o json | jq -r .data.MASTER_PASSWORD | base64 -d Log in as the Master Admin Portal administrator to verify these credentials are working. Additional resources Reference documentation Optionally, you can create new tenants on the MASTER portal URL : master.USD{wildcardDomain} . 1.4.4. External databases for 3scale API Management using the operator Important When you externalize databases from a Red Hat 3scale API Management deployment, this means to provide isolation from the application and resilience against service disruptions at the database level. The resilience to service disruptions depends on the service level agreements (SLAs) provided by the infrastructure or platform provider where you host the databases. This is not offered by 3scale. For more details on externalizing of databases offered by your chosen deployment, see the associated documentation. When you use an external databases for 3scale using the operator, the aim is to provide uninterrupted uptime if, for example, one or more databases were to fail. If you use external databases in your 3scale operator-based deployment, note the following: Configure and deploy 3scale critical databases externally. Critical databases include the system database, system redis, and backend redis components. Ensure that you deploy and configure these components in a way that makes them highly available. Specify the connection endpoints to those components for 3scale by creating their corresponding Kubernetes secrets before deploying 3scale. See External databases installation for more information. See Enabling Pod Disruption Budgets for more information about non-database deployment configurations. In the APIManager CR, set the .spec.externalComponents attribute to specify that system database, system redis, and backend redis are external: externalComponents: backend: redis: true system: database: true redis: true zync: database: true Additionally, if you want the zync database to be highly available to avoid zync potentially losing queue jobs data on restart, note the following: Deploy and configure the zync database externally. Make sure you deploy and configure the database in a way that it is highly available. Specify the connection endpoint to the zync database for 3scale by creating its corresponding Kubernetes secret before deploying 3scale. See Zync database secret . Configure 3scale by setting the .spec.externalComponents.zync.database attribute in the APIManager CR to true to specify that the zync database is an external database. 1.5. Deployment configuration options for 3scale API Management on OpenShift using the operator Important Links contained in this note to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content. This section provides information about the deployment configuration options for Red Hat 3scale API Management on OpenShift using the operator. Prerequisites Deploying 3scale API Management using the operator first requires that you follow the steps in Installing the 3scale API Management operator on OpenShift . OpenShift Container Platform 4.x. A user account with administrator privileges in the OpenShift cluster. 1.5.1. Configuring proxy parameters for embedded APIcast As a 3scale administrator, you can configure proxy parameters for embedded APIcast staging and production. This section provides reference information for specifying proxy parameters in an APIManager custom resource (CR). In other words, you are using the 3scale operator, an APIManager CR to deploy 3scale on OpenShift. You can specify these parameters when you deploy an APIManager CR for the first time or you can update a deployed APIManager CR and the operator will reconcile the update. See Deploying the APIManager custom resource . There are four proxy-related configuration parameters for embedded APIcast: allProxy httpProxy httpsProxy noProxy allProxy The allProxy parameter specifies an HTTP or HTTPS proxy to be used for connecting to services when a request does not specify a protocol-specific proxy. After you set up a proxy, configure APIcast by setting the allProxy parameter to the address of the proxy. Authentication is not supported for the proxy. In other words, APIcast does not send authenticated requests to the proxy. The value of the allProxy parameter is a string, there is no default, and the parameter is not required. Use this format to set the spec.apicast.productionSpec.allProxy parameter or the spec.apicast.stagingSpec.allProxy parameter: <scheme>://<host>:<port> For example: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: allProxy: http://forward-proxy:80 stagingSpec: allProxy: http://forward-proxy:81 httpProxy The httpProxy parameter specifies an HTTP proxy to be used for connecting to HTTP services. After you set up a proxy, configure APIcast by setting the httpProxy parameter to the address of the proxy. Authentication is not supported for the proxy. In other words, APIcast does not send authenticated requests to the proxy. The value of the httpProxy parameter is a string, there is no default, and the parameter is not required. Use this format to set the spec.apicast.productionSpec.httpProxy parameter or the spec.apicast.stagingSpec.httpProxy parameter: http://<host>:<port> For example: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: httpProxy: http://forward-proxy:80 stagingSpec: httpProxy: http://forward-proxy:81 httpsProxy The httpsProxy parameter specifies an HTTPS proxy to be used for connecting to services. After you set up a proxy, configure APIcast by setting the httpsProxy parameter to the address of the proxy. Authentication is not supported for the proxy. In other words, APIcast does not send authenticated requests to the proxy. The value of the httpsProxy parameter is a string, there is no default, and the parameter is not required. Use this format to set the spec.apicast.productionSpec.httpsProxy parameter or the spec.apicast.stagingSpec.httpsProxy parameter: https://<host>:<port> For example: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: httpsProxy: https://forward-proxy:80 stagingSpec: httpsProxy: https://forward-proxy:81 noProxy The noProxy parameter specifies a comma-separated list of hostnames and domain names. When a request contains one of these names, APIcast does not proxy the request. If you need to stop access to the proxy, for example during maintenance operations, set the noProxy parameter to an asterisk (*). This matches all hosts specified in all requests and effectively disables any proxies. The value of the noProxy parameter is a string, there is no default, and the parameter is not required. Specify a comma-separated string to set the spec.apicast.productionSpec.noProxy parameter or the spec.apicast.stagingSpec.noProxy parameter. For example: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: noProxy: theStore,company.com,big.red.com stagingSpec: noProxy: foo,bar.com,.extra.dot.com Additional resources Custom resource definition for APIcast staging configuration in an APIManager custom resource Custom resource definition for APIcast production configuration in an APIManager custom resource 1.5.2. Injecting custom environments with the 3scale API Management operator In a 3scale installation that uses embedded APIcast, you can use the 3scale operator to inject custom environments. Embedded APIcast is also referred to as managed or hosted APIcast. A custom environment defines behavior that APIcast applies to all upstream APIs that the gateway serves. To create a custom environment, define a global configuration in Lua code. You can inject a custom environment before or after 3scale installation. After injecting a custom environment and after 3scale installation, you can remove a custom environment. The 3scale operator reconciles the changes. Prerequisites The 3scale operator is installed. Procedure Write Lua code that defines the custom environment that you want to inject. For example, the following env1.lua file shows a custom logging policy that the 3scale operator loads for all services. local cjson = require('cjson') local PolicyChain = require('apicast.policy_chain') local policy_chain = context.policy_chain local logging_policy_config = cjson.decode([[ { "enable_access_logs": false, "custom_logging": "\"{{request}}\" to service {{service.id}} and {{service.name}}" } ]]) policy_chain:insert( PolicyChain.load_policy('logging', 'builtin', logging_policy_config), 1) return { policy_chain = policy_chain, port = { metrics = 9421 }, } Create a secret from the Lua file that defines the custom environment. For example: USD oc create secret generic custom-env-1 --from-file=./env1.lua A secret can contain multiple custom environments. Specify the -from-file option for each file that defines a custom environment. The operator loads each custom environment. Define an APIManager custom resource (CR) that references the secret you just created. The following example shows only content relative to referencing the secret that defines the custom environment. apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager-apicast-custom-environment spec: wildcardDomain: <desired-domain> apicast: productionSpec: customEnvironments: - secretRef: name: custom-env-1 stagingSpec: customEnvironments: - secretRef: name: custom-env-1 An APIManager CR can reference multiple secrets that define custom environments. The operator loads each custom environment. Create the APIManager CR that adds the custom environment. For example: USD oc apply -f apimanager.yaml steps You cannot update the content of a secret that defines a custom environment. If you need to update the custom environment you can do either of the following: The recommended option is to create a secret with a different name and update the APIManager CR field, customEnvironments[].secretRef.name . The operator triggers a rolling update and loads the updated custom environment. Alternatively, you can update the existing secret, redeploy APIcast by setting spec.apicast.productionSpec.replicas or spec.apicast.stagingSpec.replicas to 0, and then redeploy APIcast again by setting spec.apicast.productionSpec.replicas or spec.apicast.stagingSpec.replicas back to its value. 1.5.3. Injecting custom policies with the 3scale API Management operator In a 3scale installation that uses embedded APIcast, you can use the 3scale operator to inject custom policies. Embedded APIcast is also referred to as managed or hosted APIcast. Injecting a custom policy adds the policy code to APIcast. You can then use either of the following to add the custom policy to an API product's policy chain: 3scale API Product custom resource (CR) To use the 3scale Admin Portal to add the custom policy to a product's policy chain, you must also register the custom policy's schema with a CustomPolicyDefinition CR. Custom policy registration is a requirement only when you want to use the Admin Portal to configure a product's policy chain. You can inject a custom policy as part of or after 3scale installation. After injecting a custom policy and after 3scale installation, you can remove a custom policy by removing its specification from the APIManager CR. The 3scale operator reconciles the changes. Prerequisites You are installing or you previously installed the 3scale operator. You have defined a custom policy as described in Write your own policy . That is, you have already created, for example, the my-policy.lua , apicast-policy.json , and init.lua files that define a custom policy. Procedure Create a secret from the files that define one custom policy. For example: USD oc create secret generic my-first-custom-policy-secret \ --from-file=./apicast-policy.json \ --from-file=./init.lua \ --from-file=./my-first-custom-policy.lua If you have more than one custom policy, create a secret for each custom policy. A secret can contain only one custom policy. Use the 3scale operator to monitor secret changes. Add the apimanager.apps.3scale.net/watched-by=apimanager label to begin the 3scale operator secret changes monitoring: USD oc label secret my-first-custom-policy-secret apimanager.apps.3scale.net/watched-by=apimanager Note By default, changes to the secret are not tracked by the 3scale operator. With the label in place, the 3scale operator automatically updates the APIcast deployment whenever you make changes to the secret. This happens in both staging and production environments where the secret is in use. The 3scale operator will not take ownership of the secret in any way. Define an APIManager CR that references each secret that contains a custom policy. You can specify the same secret for APIcast staging and APIcast production. The following example shows only content relative to referencing secrets that contain custom policies. apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager-apicast-custom-policy spec: apicast: stagingSpec: customPolicies: - name: my-first-custom-policy version: "0.1" secretRef: name: my-first-custom-policy-secret - name: my-second-custom-policy version: "0.1" secretRef: name: my-second-custom-policy-secret productionSpec: customPolicies: - name: my-first-custom-policy version: "0.1" secretRef: name: my-first-custom-policy-secret - name: my-second-custom-policy version: "0.1" secretRef: name: my-second-custom-policy-secret An APIManager CR can reference multiple secrets that define different custom policies. The operator loads each custom policy. Create the APIManager CR that references the secrets that contain the custom policies. For example: USD oc apply -f apimanager.yaml steps When you apply the apimanager.apps.3scale.net/watched-by=apimanager label, the 3scale operator begins monitoring changes in the secret. Now, you can modify the custom policy within the secret, and the operator will initiate a rolling update, loading the updated custom environment. Alternatively, you can update the existing secret, redeploy APIcast by setting spec.apicast.productionSpec.replicas or spec.apicast.stagingSpec.replicas to 0, and then redeploy APIcast again by setting spec.apicast.productionSpec.replicas or spec.apicast.stagingSpec.replicas back to its value. 1.5.4. Configuring OpenTracing with the 3scale API Management operator In a 3scale installation that uses embedded APIcast, you can use the 3scale operator to configure OpenTracing. You can configure OpenTracing in the staging or production environments or both environments. By enabling OpenTracing, you get more insight and better observability on the APIcast instance. Prerequisites The 3scale operator is installed or you are in the process of installing it. Procedure Define a secret that contains your OpenTracing configuration details in stringData.config . This is the only valid value for the attribute that contains your OpenTracing configuration details. Any other specification prevents APIcast from receiving your OpenTracing configuration details. The following example shows a valid secret definition: apiVersion: v1 kind: Secret metadata: name: myjaeger stringData: config: |- { "service_name": "apicast", "disabled": false, "sampler": { "type": "const", "param": 1 }, "reporter": { "queueSize": 100, "bufferFlushInterval": 10, "logSpans": false, "localAgentHostPort": "jaeger-all-in-one-inmemory-agent:6831" }, "headers": { "jaegerDebugHeader": "debug-id", "jaegerBaggageHeader": "baggage", "TraceContextHeaderName": "uber-trace-id", "traceBaggageHeaderPrefix": "testctx-" }, "baggage_restrictions": { "denyBaggageOnInitializationFailure": false, "hostPort": "127.0.0.1:5778", "refreshInterval": 60 } } type: Opaque Create the secret. For example, if you saved the secret definition in the myjaeger.yaml file, you would run the following command: USD oc create -f myjaeger.yaml Define an APIManager custom resource (CR) that specifies OpenTracing attributes. In the CR definition, set the openTracing.tracingConfigSecretRef.name attribute to the name of the secret that contains your OpenTracing configuration details. The following example shows only content relative to configuring OpenTracing: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager1 spec: apicast: stagingSpec: ... openTracing: enabled: true tracingLibrary: jaeger tracingConfigSecretRef: name: myjaeger productionSpec: ... openTracing: enabled: true tracingLibrary: jaeger tracingConfigSecretRef: name: myjaeger Create the APIManager CR that configures OpenTracing. For example, if you saved the APIManager CR in the apimanager1.yaml file, you would run the following command: USD oc apply -f apimanager1.yaml steps Depending on how OpenTracing is installed, you should see the traces in the Jaeger service user interface. Additional resource APIManager custom resource definition 1.5.5. Enabling TLS at the pod level with the 3scale API Management operator 3scale deploys two APIcast instances, one for production and the other for staging. TLS can be enabled for only production or only staging, or for both instances. Prerequisites A valid certificate for enabling TLS. Procedure Create a secret from your valid certificate, for example: USD oc create secret tls mycertsecret --cert=server.crt --key=server.key The configuration exposes secret references in the APIManager custom resource (CR). You create the secret and then reference the name of the secret in the APIManager CR as follows: Production: The APIManager CR exposes the certificate in the .spec.apicast.productionSpec.httpsCertificateSecretRef field. Staging: The APIManager CR exposes the certificate in the .spec.apicast.stagingSpec.httpsCertificateSecretRef field. Optionally, you can configure the following: httpsPort indicates which port APIcast should start listening on for HTTPS connections. If this clashes with the HTTP port APIcast uses this port for HTTPS only. httpsVerifyDepth defines the maximum length of the client certificate chain. Note Provide a valid certificate and reference from the APIManager CR. If the configuration can access httpsPort but not httpsCertificateSecretRef , APIcast uses an embedded self-signed certificate. This is not recommended. Click Operators > Installed Operators . From the list of Installed Operators , click 3scale Operator . Click the API Manager tab. Click Create APIManager . Add the following YAML definitions to the editor. If enabling for production , configure the following YAML definitions: spec: apicast: productionSpec: httpsPort: 8443 httpsVerifyDepth: 1 httpsCertificateSecretRef: name: mycertsecret If enabling for staging , configure the following YAML defintions: spec: apicast: stagingSpec: httpsPort: 8443 httpsVerifyDepth: 1 httpsCertificateSecretRef: name: mycertsecret Click Create . 1.5.6. Proof of concept for evaluation deployment The following sections describe the configuration options applicable to the proof of concept for an evaluation deployment of 3scale. This deployment uses internal databases as default. Important The configuration for external databases is the standard deployment option for production environments. 1.5.6.1. Default deployment configuration Containers will have Kubernetes resource limits and requests . This ensures a minimum performance level. It limits resources to allow external services and allocation of solutions. Deployment of internal databases. File storage will be based on Persistence Volumes (PV). One will require read, write, execute (RWX) access mode. OpenShift configured to provide them upon request. Deploy MySQL as the internal relational database. The default configuration option is suitable for proof of concept (PoC) or evaluation by a customer. One, many, or all of the default configuration options can be overridden with specific field values in the APIManager custom resource (CR). The 3scale operator allows all available combinations. For example, the 3scale operator allows deployment of 3scale in evaluation mode and external databases mode. 1.5.6.2. Evaluation installation For and evaluation installation, containers will not have kubernetes resource limits and requests specified. For example: Small memory footprint Fast startup Runnable on laptop Suitable for presale/sales demos apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: lvh.me resourceRequirementsEnabled: false Additional resources APIManager 1.5.7. External databases installation Important When you externalize databases from a Red Hat 3scale API Management deployment, this means to provide isolation from the application and resilience against service disruptions at the database level. The resilience to service disruptions depends on the service level agreements (SLAs) provided by the infrastructure or platform provider where you host the databases. This is not offered by 3scale. For more details on externalizing of databases offered by your chosen deployment, see the associated documentation. Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content. An external databases installation is suitable for production where you want to provide uninterrupted uptime or where you plan to reuse your own databases. Important When enabling the 3scale external databases installation mode, you can configure one or more of the following databases as external to 3scale: backend-redis system-redis system-database ( mysql , postgresql , or oracle ) zync-database Before creating an APIManager CR to deploy 3scale, you must provide the following connection settings for the external databases by using OpenShift secrets. Additional resources Red Hat 3scale API Management Supported Configurations 1.5.7.1. Backend Redis secret 3scale backend uses two Redis databases: one for storage and one for queues. The storage database holds information about API services, application keys, metrics, limits, and usage data, including current usage and historical analytics. Utilization data is unique to the Redis storage database, while the database of the System component is the main source for other data, which you can recreate if needed. The queues database temporarily stores background job queues, based on Resque , for reporting API usage. The backend listener creates these jobs, and the backend worker processes them. Deploy two external Redis instances and fill in the connection settings as shown in the following example: apiVersion: v1 kind: Secret metadata: name: backend-redis stringData: REDIS_STORAGE_URL: "redis://backend-redis-storage" REDIS_STORAGE_SENTINEL_HOSTS: "redis://sentinel-0.example.com:26379,redis://sentinel-1.example.com:26379, redis://sentinel-2.example.com:26379" REDIS_STORAGE_SENTINEL_ROLE: "master" REDIS_QUEUES_URL: "redis://backend-redis-queues" REDIS_QUEUES_SENTINEL_HOSTS: "redis://sentinel-0.example.com:26379,redis://sentinel-1.example.com:26379, redis://sentinel-2.example.com:26379" REDIS_QUEUES_SENTINEL_ROLE: "master" type: Opaque The Secret name must be backend-redis . Table 1.1. backend-redis Field Description Default value REDIS_STORAGE_URL Backend Redis storage database URL. Mandatory when the instance is managed externally. Otherwise the default value is: redis://backend-redis:6379/0 REDIS_STORAGE_SENTINEL_ROLE Backend Redis Storage Sentinel Role name ( master or slave ). Use only when Redis Sentinel is configured in the Redis database. "" REDIS_STORAGE_SENTINEL_HOSTS Backend Redis Storage Sentinel Hosts name. Use only when Redis Sentinel is configured in the Redis database. "" REDIS_QUEUES_URL Backend Redis Queues database URL. Mandatory when the instance is managed externally. Otherwise the default value is: redis://backend-redis:6379/1 REDIS_QUEUES_SENTINEL_ROLE Backend Redis Queues Sentinel Role name ( master or slave ). Use only when Redis Sentinel is configured in the Redis database. "" REDIS_QUEUES_SENTINEL_HOSTS Backend Redis Queues Sentinel Hosts name. Use only when Redis Sentinel is configured in the Redis database. "" 1.5.7.2. System Redis secret Deploy two external Redis instances and fill in the connection settings as shown in the following example: apiVersion: v1 kind: Secret metadata: name: system-redis stringData: URL: "redis://system-redis" SENTINEL_HOSTS: "redis://sentinel-0.example.com:26379,redis://sentinel-1.example.com:26379, redis://sentinel-2.example.com:26379" SENTINEL_ROLE: "master" type: Opaque The Secret name must be system-redis . Table 1.2. system-redis Field Description Default value URL System Redis database URL. Mandatory when instance is managed externally. Otherwise the default value is: redis://system-redis:6379/1 SENTINEL_HOSTS System Redis Sentinel hosts. Use only when Redis Sentinel is configured. "" SENTINEL_ROLE System Redis Sentinel role name ( master or slave ). Use only when Redis Sentinel is configured. "" 1.5.7.3. System database secret Note The Secret name must be system-database . When you are deploying 3scale, you have three alternatives for your system database. Configure different attributes and values for each alternative's related secret. MySQL PostgreSQL Oracle Database To deploy a MySQL, PostgreSQL, or an Oracle Database system database secret, fill in the connection settings as shown in the following examples: MySQL system database secret apiVersion: v1 kind: Secret metadata: name: system-database stringData: URL: "mysql2://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}" type: Opaque Important If you use MySQL 8.0 with 3scale 2.15, you must set the authentication plugin to mysql_native_password . Add the following to the MySQL configuration file: PostgreSQL system database secret apiVersion: v1 kind: Secret metadata: name: system-database stringData: URL: "postgresql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}" type: Opaque Oracle system database secret apiVersion: v1 kind: Secret metadata: name: system-database stringData: URL: "oracle-enhanced://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}" ORACLE_SYSTEM_PASSWORD: "{SYSTEM_PASSWORD}" type: Opaque Note {DB_USER} and {DB_PASSWORD} are the username and password of the regular non-system user. {DB_NAME} is the Oracle Database service name . ORACLE_SYSTEM_PASSWORD is optional, see Configure a database user . 1.5.7.4. Zync database secret In a zync database setup, when the spec.externalComponents.zync.database field is set to true , you must create a secret named zync before you deploy 3scale. In this secret, set the DATABASE_URL and DATABASE_PASSWORD fields to the values that point to your external zync database, for example: apiVersion: v1 kind: Secret metadata: name: zync stringData: DATABASE_URL: postgresql://<zync-db-user>:<zync-db-password>@<zync-db-host>:<zync-db-port>/zync_production ZYNC_DATABASE_PASSWORD: <zync-db-password> type: Opaque The zync database must be in high-availability mode. 1.5.7.5. APIManager custom resources to deploy 3scale API Management Note When you enable external components, you must create a secret for each external component ( backend-redis , system-redis , system-database , zync ) before you deploy 3scale. For an external system-database , choose only one type of database to externalize. Configuration of the APIManager custom resource (CR) depends on whether or not your choice of database is external to your 3scale deployment. If backend-redis , system-redis , or system-database is external to 3scale, populate the APIManager CR externalComponents object as shown in the following example: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: lvh.me externalComponents: system: database: true Additional resources Backend redis secret System database secret APIManager ExternalComponentsSpec Zync secret 1.5.8. Enabling pod affinity in the 3scale API Management operator You can enable pod affinities in the 3scale operator for every component. This ensures distribution of pod replicas from each deploymentConfig across different nodes of the cluster, so they will be evenly balanced across different availability zones (AZ). 1.5.8.1. Customizing node affinity and tolerations at component level Customize kubernetes affinity and tolerations in your 3scale solution through the APIManager CR attributes. You can then customize to schedule different 3scale components onto kubernetes nodes. For example, to set a custom node affinity for backend-listener and custom tolerations for system-memcached , do the following: Custom affinity and tolerations apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: backend: listenerSpec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/hostname" operator: In values: - ip-10-96-1-105 - key: "beta.kubernetes.io/arch" operator: In values: - amd64 system: memcachedTolerations: - key: key1 value: value1 operator: Equal effect: NoSchedule - key: key2 value: value2 operator: Equal effect: NoSchedule Add the following affinity block to apicastProductionSpec or to any non-database deploymentConfig . This adds a soft podAntiAffinity configuration using preferredDuringSchedulingIgnoredDuringExecution . The scheduler will try to run this set of apicast-production pods in different hosts from different AZs. If it is not possible, then allow them to run elsewhere: Soft podAntiAffinity affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: deploymentConfig: apicast-production topologyKey: kubernetes.io/hostname - weight: 99 podAffinityTerm: labelSelector: matchLabels: deploymentConfig: apicast-production topologyKey: topology.kubernetes.io/zone In the following example, a hard podAntiAffinity configuration is set using requiredDuringSchedulingIgnoredDuringExecution . Conditions must be met to schedule a pod onto a node. A risk exists, for example, that you will not be able to schedule new pods on a cluster with low free resources: Hard podAntiAffinity affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: deploymentConfig: apicast-production topologyKey: kubernetes.io/hostname - weight: 99 podAffinityTerm: labelSelector: matchLabels: deploymentConfig: apicast-production topologyKey: topology.kubernetes.io/zone Additional resources APIManager CDR reference 1.5.9. Multiple clusters in multiple availability zones Note In case of failure, bringing a passive cluster into active mode disrupts the provision of the service until the procedure finishes. Due to this disruption, be sure to have a maintenance window. This documentation focuses on deployment using Amazon Web Services (AWS). The same configuration options apply to other public cloud vendors where the provider's managed database services offer, for example, support for multiple availability zones and multiple regions. When you want to install 3scale on several OpenShift clusters and high availability (HA) zones, there are options available which you can refer to here. In multiple cluster installation options, clusters work in an active/passive configuration, with the failover procedure involving a few manual steps. 1.5.9.1. Prerequisites for multiple clusters installations Use the following in 3scale installations that involve using several OpenShift clusters: Use pod affinities with both kubernetes.io/hostname and topology.kubernetes.io/zone rules in the APIManager custom resource (CR). Use pod affinities with both kubernetes.io/hostname and topology.kubernetes.io/zone rules in the APIManager custom resource (CR). Use pod disruption budgets in the APIManager CR. A 3scale installation over multiple clusters must use the same shared wildcardDomain attribute specifications in the APIManager CR. The use of a different domain for each cluster is not allowed in this installation mode, as the information stored in the database would be conflicting. You must manually deploy the secrets containing credentials, such as tokens and passwords, in all clusters with the same values. The 3scale operator creates them with secure random values on every cluster. In this case, you need to have the same credentials in both clusters. You will find the list of secrets and how to configure them in the 3scale operator documentation. The following is the list of secrets you must mirror in both clusters: backend-internal-api system-app system-events-hook system-master-apicast system-seed You must manually deploy secrets with the database connection strings for backend-redis , system-redis , system-database and zync . See External databases installation . Databases shared among clusters must use the same values on all clusters. If each cluster have their own databases, they must use different values on each cluster. 1.5.9.2. Active-passive clusters on the same region with shared databases This setup consists of having two or more clusters in the same region and deploying 3scale in active-passive mode. One cluster is active, receiving traffic. The others are in standby mode without receiving traffic, therefore passive, but prepared to assume the active role in case there is a failure in the active cluster. In this installation option, only a single region is in use and databases will be shared among all clusters. 1.5.9.3. Configuring and installing shared databases Procedure Create two or more OpenShift clusters in the same region using different availability zones (AZs). A minimum of three zones is recommended. Create all required AWS ElastiCache (EC) instances with Amazon Relational Database Service (RDS) Multi-AZ enabled: One AWS EC for Backend Redis database One AWS EC for System Redis database Create all required AWS RDS instances with Amazon RDS Multi-AZ enabled: One AWS RDS for the System database One AWS RDS for Zync database Configure a AWS S3 bucket for the system assets. Create a custom domain in AWS Route53 or your DNS provider and point it to the OpenShift router of the active cluster. This must coincide with the wildcardDomain attribute from APIManager custom resource (CR). Install 3scale in the passive cluster. The APIManager CR should be identical to the one used in the step. When all pods are running, change the APIManager to deploy 0 replicas for all the backend , system , zync and APIcast pods. Set replicas to 0 to avoid consuming jobs from active database. Deployment will fail due to pod dependencies if each replica is set to 0 at first. For example, pods checking that others are running. First deploy as normal, then set replicas to 0 as shown in the APIManager spec example: spec: apicast: stagingSpec: replicas: 0 productionSpec: replicas: 0 backend: listenerSpec: replicas: 0 workerSpec: replicas: 0 cronSpec: replicas: 0 zync: appSpec: replicas: 0 queSpec: replicas: 0 system: appSpec: replicas: 0 sidekiqSpec: replicas: 0 1.5.9.4. Manual failover shared databases Procedure In the active cluster, scale down the replicas of the backend , system , zync , and APIcast pods to 0. This becomes the new passive cluster, so you ensure that the new passive cluster will not consume jobs from active databases. Downtime starts here. In the passive cluster, edit the APIManager to scale up the replicas of the backend , system , zync , and APIcast pods that were set to 0, so it will become the active cluster. In the new active cluster, recreate the OpenShift routes created by zync . Run the zync:resync:domains command from the system-master container of the system-app pod: bundle exec rake zync:resync:domains Point the custom domain created in AWS Route53 to the OpenShift router of the new active cluster. The old passive cluster will start to receive traffic, and becomes the new active cluster. 1.5.9.5. Active-passive clusters on different regions with synced databases This setup consists of having two or more clusters in different regions and deploying 3scale in active-passive mode. One cluster is active, receiving traffic, the others are in standby mode without receiving traffic, therefore passive, but prepared to assume the active role in case there is a failure in the active cluster. To ensure good database access latency, each cluster has its own database instances. The databases from the active 3scale installation are replicated to the read-replica databases of the 3scale passive installations so the data is available and up to date in all regions for a possible failover. 1.5.9.6. Configuring and installing synced databases Procedure Create two or more OpenShift clusters in different regions using different availability zones. A minimum of three zones is recommended. Create all required AWS ElastiCache instances with Amazon RDS Multi-AZ enabled on every region: Two AWS EC for Backend Redis database: one per region. Two AWS EC for System Redis database: one per region. Use the cross-region replication with the Global Datastore feature enabled, so the databases from passive regions are read-replicas from the master databases at the active region. Create all required AWS RDS instances with Amazon RDS Multi-AZ enabled on every region: Two AWS RDS for the System database. Two AWS RDS for Zync database. Use cross-region replication, so the databases from passive regions are read-replicas from the master databases at the active region. Configure a AWS S3 bucket for the system assets on every region using cross-region replication. Create a custom domain in AWS Route53 or your DNS provider and point it to the OpenShift Router of the active cluster. This must coincide with the wildcardDomain attribute from APIManager CR. Install 3scale in the passive cluster. The APIManager CR should be identical to the one used in the step. When all pods are running, change the APIManager to deploy 0 replicas for all the backend , system , zync , and APIcast pods. Set replicas to 0 to avoid consuming jobs from active database. Deployment will fail due to pod dependencies if each replica is set to 0 at first. For example, pods checking that others are running. First deploy as normal, then set replicas to 0. 1.5.9.7. Manual failover synced databases Procedure Do steps 1, 2 and 3 from Manual Failover shared databases . Every cluster has its own independent databases: read-replicas from the master at the active region. You must manually execute a failover on every database to select the new master on the passive region, which then becomes the active region. Manual failovers of the databases to execute are: AWS RDS: System and Zync . AWS ElastiCaches: Backend and System . Do step 4 from Manual Failover shared databases . 1.5.10. Amazon Simple Storage Service 3scale API Management fileStorage installation Before creating APIManager custom resource (CR) to deploy 3scale, provide connection settings for the Amazon Simple Storage Service (Amazon S3) service by using an OpenShift secret. Important Skip this section if you are deploying 3scale with the local filesystem storage. The name you choose for a secret can be any name as long as it is not an existing secret name and it will be referenced in the APIManager CR. If AWS_REGION is not provided for S3 compatible storage, use default or the deployment will fail. Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content. 1.5.10.1. Amazon S3 bucket creation Prerequisites You must have an Amazon Web Services (AWS) account. Procedure Create a bucket for storing the system assets. Disable the public access blocker of S3 when using the Logo feature of the Developer Portal. Create an Identity and Access Management (IAM) policy with the following minimum permissions: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::<target_bucket_name>", 1 "arn:aws:s3:::<target_bucket_name>/*" 2 ] } ] } 1 Replace <target_bucket_name> with your own value. 2 Replace <target_bucket_name> with your own value. Create a CORS configuration with the following rules: 1.5.10.2. Create an OpenShift secret The following examples show 3scale fileStorage using Amazon S3 instead of persistent volume claim (PVC). Note AN AWS S3 compatible provider can be configured in the S3 secret with AWS_HOSTNAME , AWS_PATH_STYLE , and AWS_PROTOCOL optional keys. See the fileStorage S3 credentials secret fields table for more details. In the following example, Secret name can be anything, as it is be referenced in the APIManager CR. apiVersion: v1 kind: Secret metadata: creationTimestamp: null name: aws-auth stringData: AWS_ACCESS_KEY_ID: <ID_123456> AWS_SECRET_ACCESS_KEY: <ID_98765544> AWS_BUCKET: <mybucket.example.com> AWS_REGION: eu-west-1 type: Opaque Lastly, create the APIManager CR to deploy 3scale. apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: <example_apimanager> namespace: <3scale_test> spec: wildcardDomain: lvh.me system: fileStorage: simpleStorageService: configurationSecretRef: name: aws-auth Check APIManager SystemS3Spec . The following table shows the fileStorage Amazon S3 credentials secret field requirements for Identity and Access Management (IAM) and Security Token Service (STS) settings: The S3 authentication method using Secure Token Service (STS) is for short-term, limited-privilege security credentials. S3 Identity and Access Management (IAM) is for long-term privilege security credentials. Table 1.3. fileStorage S3 credentials secret fields Field Description Required for IAM Required for STS AWS_ACCESS_KEY_ID AWS Access Key ID to use in S3 Storage for system's fileStorage Yes No AWS_SECRET_ACCESS_KEY AWS Access Key Secret to use in S3 Storage for system's fileStorage Yes No AWS_BUCKET The S3 bucket to be used as system's fileStorage for assets Yes Yes AWS_REGION The region of the S3 bucket to be used as system's fileStorage for assets Yes Yes AWS_HOSTNAME Default: Amazon endpoints - An AWS S3 compatible provider endpoint hostname No No AWS_PROTOCOL Default: HTTPS An AWS S3 compatible provider endpoint protocol No No AWS_PATH_STYLE Default: false When set to true , the bucket name is always left in the request URI and never moved to the host as a sub-domain No No AWS_ROLE_ARN ARN of the Role which has a policy attached to authenticate using AWS STS No Yes AWS_WEB_IDENTITY_TOKEN_FILE Path to mounted token file location. For example: /var/run/secrets/openshift/serviceaccount/token No Yes 1.5.10.3. Manual mode with STS STS authentication mode must be enabled from the APIManager CR. You can define your audience, however, the default value is openshift . Prerequisites Configure OpenShift to use temporary credentials for different components with AWS Security Token Service (STS). For further detail see Using manual mode with Amazon Web Services Secure Token Service . apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: <apimanager_sample> namespace: <3scale_test> spec: wildcardDomain: lvh.me system: fileStorage: simpleStorageService: configurationSecretRef: name: s3-credentials sts: enabled: true audience: openshift The secret generated by the cloud credential tooling looks different from the IAM secret. There are two new fields AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE instead of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY . STS secret example kind: Secret apiVersion: v1 metadata: name: s3-credentials namespace: 3scale data: AWS_ROLE_ARN: arn:aws:iam::ID:role/ROLE AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/openshift/serviceaccount/token AWS_BUCKET: <mybucket.example.com> AWS_REGION: eu-west-1 type: Opaque With STS, the 3scale operator adds the projected volume to request the token. The following pods have a projected volume: system-app system-app hook pre system-sidekiq Pod example for STS apiVersion: v1 kind: Pod metadata: name: system-sidekiq-1-zncrz namespace: 3scale-test spec: containers: .... volumeMounts: - mountPath: /var/run/secrets/openshift/serviceaccount name: s3-credentials readOnly: true .... volumes: - name: s3-credentials projected: defaultMode: 420 sources: - serviceAccountToken: audience: openshift expirationSeconds: 3600 path: token Additional resources APIManager SystemS3Spec S3 secret reference Using manual mode with Amazon Web Services Secure Token Service Short lived Credentials with AWS Security Token Service 1.5.11. PostgreSQL installation A MySQL internal relational database is the default deployment. This deployment configuration can be overridden to use PostgreSQL instead. apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: lvh.me system: database: postgresql: {} Additional resources APIManager DatabaseSpec 1.5.12. Configuring SMTP variables (optional) 3scale uses email to send notifications and invite new users . If you intend to use these features, you must provide your own SMTP server and configure SMTP variables in the system-smtp secret. Procedure If you are not already logged in, log in to OpenShift and select the project where your 3scale On-premises instance is installed using the oc command: USD oc login -u <user> <url> USD oc project <3scale-project> Use the oc patch command to update the secret system-smtp . Use the option -p with the following JSON: {"stringData":{"<field>":"<value>"}} , where <field> is the field of the secret, and <value> is the desired value. The complete list of available fields is listed in the table below. Table 1.4. system-smtp Field Description Default value in the secret Default value in the application mailer address The address of a remote mail server. "" nil port The port of the remote mail server to use. "" 0 username Username if the mail server requires authentication and the authentication type requires it. "" nil password Password if the mail server requires authentication and the authentication type requires it. "" nil authentication Use if the mail server requires authentication. Set the authentication types: plain to send the password in the clear, login to send password Base64 encoded, or cram_md5 to combine a challenge/response mechanism based on the HMAC-MD5 algorithm. "" nil openssl.verify.mode When using TLS, you can set how OpenSSL checks the certificate. This is useful if you need to validate a self-signed and/or a wildcard certificate. You can use the name of an OpenSSL verify constant: none or peer . "" nil from_address from address value for the no-reply mail. "" "no-reply@{wildcardDomain}" Note HELO domain used for the mail server configuration is the value of {wildcardDomain} . {wildcardDomain} is a placeholder for the value that is set in spec.wildcardDomain in the APIManager CR. Examples USD oc patch secret system-smtp -p '{"stringData":{"address":"<your_address>"}}' USD oc patch secret system-smtp -p '{"stringData":{"port":"587"}}' USD oc patch secret system-smtp -p '{"stringData":{"username":"<your_username>"}}' USD oc patch secret system-smtp -p '{"stringData":{"password":"<your_password>"}}' USD oc patch secret system-smtp -p '{"stringData":{"authentication":"plain"}}' USD oc patch secret system-smtp -p '{"stringData":{"openssl.verify.mode":"none"}}' After you have set the secret variables, redeploy the system-app and system-sidekiq pods: USD oc rollout restart deployment/system-app USD oc rollout restart deployment/system-sidekiq Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/system-app USD oc rollout status deployment/system-sidekiq 1.5.13. Customizing compute resource requirements at component level Customize Kubernetes Compute Resource Requirements in your 3scale solution through the APIManager custom resource (CR) attributes. Do this to customize compute resource requirements, which is CPU and memory, assigned to a specific APIManager component. The following example outlines how to customize compute resource requirements for the system-master's system-provider container, for the backend-listener and for the zync-database : apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: backend: listenerSpec: resources: requests: memory: "150Mi" cpu: "300m" limits: memory: "500Mi" cpu: "1000m" system: appSpec: developerContainerResources: limits: cpu: 1500m memory: 1400Mi requests: cpu: 150m memory: 600Mi masterContainerResources: limits: cpu: 1500m memory: 1400Mi requests: cpu: 150m memory: 600Mi providerContainerResources: limits: cpu: 1500m memory: 1400Mi requests: cpu: 150m memory: 600Mi zync: databaseResources: requests: memory: "111Mi" cpu: "222m" limits: memory: "333Mi" cpu: "444m" Additional resources APIManager CRD reference 1.5.13.1. Default APIManager components compute resources When you configure the APIManager spec.resourceRequirementsEnabled attribute as true , the default compute resources are set for the APIManager components. The specific compute resources default values that are set for the APIManager components are shown in the following table. 1.5.13.1.1. CPU and memory units The following list explains the units you will find mentioned in the compute resources default values table. For more information on CPU and memory units, see Managing Resources for Containers . Resource units explanation m - milliCPU or millicore Mi - mebibytes Gi - gibibyte G - gigabyte Table 1.5. Compute resources default values Component CPU requests CPU limits Memory requests Memory limits system-app's system-master 50m 1000m 600Mi 800Mi system-app's system-provider 50m 1000m 600Mi 800Mi system-app's system-developer 50m 1000m 600Mi 800Mi system-sidekiq 100m 1000m 500Mi 2Gi system-searchd 80m 1000m 250Mi 512Mi system-redis 150m 500m 256Mi 32Gi system-mysql 250m No limit 512Mi 2Gi system-postgresql 250m No limit 512Mi 2Gi backend-listener 500m 1000m 550Mi 700Mi backend-worker 150m 1000m 50Mi 300Mi backend-cron 100m 500m 100Mi 500Mi backend-redis 1000m 2000m 1024Mi 32Gi apicast-production 500m 1000m 64Mi 128Mi apicast-staging 50m 100m 64Mi 128Mi zync 150m 1 250M 512Mi zync-que 250m 1 250M 512Mi zync-database 50m 250m 250M 2G 1.5.14. Pod priority of 3scale API Management components As a 3scale administrator, you can set up the pod priority for various 3scale installed components by modifying the APIManager custom resource (CR). Use the optional priorityClassName available in the following components: apicast-production apicast-staging backend-cron backend-listener backend-worker backend-redis system-app system-sidekiq system-searchd system-memcache system-mysql system-postgresql system-redis zync zync-database zync-que For example: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: api.vmogilev01.0nno.s1.devshift.org resourceRequirementsEnabled: false apicast: stagingSpec: priorityClassName: openshift-user-critical productionSpec: priorityClassName: openshift-user-critical backend: listenerSpec: priorityClassName: openshift-user-critical cronSpec: priorityClassName: openshift-user-critical workerSpec: priorityClassName: openshift-user-critical redisPriorityClassName: openshift-user-critical system: appSpec: priorityClassName: openshift-user-critical sidekiqSpec: priorityClassName: openshift-user-critical searchdSpec: priorityClassName: openshift-user-critical searchdSpec: priorityClassName: openshift-user-critical memcachedPriorityClassName: openshift-user-critical redisPriorityClassName: openshift-user-critical database: postgresql: priorityClassName: openshift-user-critical zync: appSpec: priorityClassName: openshift-user-critical queSpec: priorityClassName: openshift-user-critical databasePriorityClassName: openshift-user-critical 1.5.15. Setting custom labels You can customize labels through the APIManager CR labels attribute for each DeploymentConfig (DC) that are applied to their respective pods. Note If you remove a label defined in a Custom Resource (CR), it is not automatically removed from the associated DeploymentConfig (DC). You must manually remove the label from the DC. Example for apicast-staging and backend-listener : apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: example.com resourceRequirementsEnabled: false backend: listenerSpec: labels: backendLabel1: sample-label1 backendLabel2: sample-label2 apicast: stagingSpec: labels: apicastStagingLabel1: sample-label1 apicastStagingLabel2: sample-label2 Additional resources APIManager CRD reference 1.5.16. Setting backend client to skip certificate verification When a controller processes an object, it generates a new backend client for making API calls. By default, this client is set up to confirm the server's certificate chain. However, during development and testing, you might need the client to skip certificate verification when processing an object. To achieve this, add the annotation "insecure_skip_verify": "true" to the following objects: ActiveDoc Application Backend CustomPolicyDefinition DeveloperAccount DeveloperUser OpenAPI - backend and product Product ProxyConfigPromote Tenant OpenAPI CR example: apiVersion: capabilities.3scale.net/v1beta1 kind: OpenAPI metadata: name: ownertest namespace: threescale annotations: "insecure_skip_verify": "true" spec: openapiRef: secretRef: name: myopenapi namespace: threescale productSystemName: testProduct 1.5.17. Setting custom annotations In 3scale, the components' pods have annotations. These are key/value pairs used for configurations. You can change these annotations for any 3scale component using the APIManager CR. Note If you remove an annotation defined in a custom resource (CR), it is not automatically removed from the associated DeploymentConfig (DC). You must manually remove the annotation from the DC. APIManager annotations for apicast-staging and backend-listener apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: example.com apicast: stagingSpec: annotations: anno-sample1: anno1 backend: listenerSpec: annotations: anno-sample2: anno2 Additional resources APIManager CRD reference 1.5.18. Reconciliation Once 3scale has been installed, the 3scale operator enables updating a given set of parameters from the custom resource (CR) to modify system configuration options. Modifications are made by hot swapping , that is, without stopping or shutting down the system. When a reconciliation event happens in the 3scale operator and the APIcast operator, there are two possible scenarios: When there is no deploymentconfig and the CR has replicas, the deploymentconfig value will match those replicas. If the CR does not contain replicas, the deploymentconfig replica value will be set to 1 . When there is a deploymentconfig and the CR has replicas, the deploymentconfig value will match those replicas, even if it is 0 . If the CR does not contain replicas, the deploymentconfig value stays the same. Not all the parameters of the APIManager CR definitions (CRDs) are reconcilable. The following is a list of reconcilable parameters: Resources Backend replicas APIcast replicas System replicas Zync replicas 1.5.18.1. Resources Resource limits and requests for all 3scale components. apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: resourceRequirementsEnabled: true/false 1.5.18.2. Backend replicas Backend components pod count. apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: backend: listenerSpec: replicas: X workerSpec: replicas: Y cronSpec: replicas: Z Note When the replica field is not set, the operator does not reconcile replicas. This allows third party controllers to manage replicas, like HorizontalPodAutoscaler controllers. It also allows update them manually on the deployment object. 1.5.18.3. APIcast replicas APIcast staging and production components pod count. apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: replicas: X stagingSpec: replicas: Z Note When the replica field is not set, the operator does not reconcile replicas. This allows third party controllers to manage replicas, like HorizontalPodAutoscaler controllers. It also allows update them manually on the deployment object. 1.5.18.4. System replicas System app and system sidekiq components pod count apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: system: appSpec: replicas: X sidekiqSpec: replicas: Z Note When the replica field is not set, the operator does not reconcile replicas. This allows third party controllers to manage replicas, like HorizontalPodAutoscaler controllers. It also allows update them manually on the deployment object. 1.5.18.5. Zync replicas Zync app and que components pod count apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: zync: appSpec: replicas: X queSpec: replicas: Z Note When the replica field is not set, the operator does not reconcile replicas. This allows third party controllers to manage replicas, like HorizontalPodAutoscaler controllers. It also allows update them manually on the deployment object. 1.5.19. Setting the APICAST_SERVICE_CACHE_SIZE environment variable You can specify the number of services that APIcast stores in the internal cache by adding an optional field in the APIManager custom resource definition (CRD). Prerequisites You have installed the APIcast operator, or you are in the process of installing it. Procedure Add the serviceCacheSize optional fields in both the production and staging sections of the spec : Verification Type the following commands to check the deployment: Verify inclusion of the environment variables: Note You can specify the number of services that APIcast stores in the internal cache by adding an optional field in the APIManager custom resource definition (CRD). When the replica field is not set, the operator does not reconcile replicas. This allows third party controllers to manage replicas, like HorizontalPodAutoscaler controllers. It also allows update them manually on the deployment object. Additional resource APIcast custom resource definition 1.6. Configuring Horizontal Pod Autoscaling for 3scale API Management Note Default values, especially thresholds, might not be sufficient for your environment and could require adjustment. Requests and limits might also need modification. Performance testing is highly recommended. Configuring Horizontal Pod Autoscaling (HPA) for Red Hat 3scale API Management components ensures optimal resource utilization and scalability. With this guide, you can enable and configure HPA for apicast-production , backend-listener , and backend-worker components. You can maintain efficient performance and handle varying workloads dynamically. Check the prerequisites before updating the APIManager custom resource (CR), and modifying the HPA configuration to suit your specific needs. Prerequisites Ensure Redis is running in async mode. This is enabled by default by the 3scale operator. ResourceRequirementsEnabled must be set to true for HPA to function. Procedure Enable HPA for apicast-production , backend-listener , and backend-worker components. Accept the default HPA configuration, which sets 85% resource utilization with a minimum of 1 pod and a maximum of 5 pods. Add the HPA configuration to your APIManager CR. The following YAML is an example configuration for backend-worker , backend-listener , and apicast-production components: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: example.com resourceRequirementsEnabled: true apicast: productionSpec: hpa: true backend: listenerSpec: hpa: true workerSpec: hpa: true Review the default HPA configuration. The default HPA configuration creates HPA instances with the following settings for backend-worker : apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: backend-worker spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: backend-worker minReplicas: 1 maxReplicas: 5 metrics: - type: Resource resource: name: cpu target: averageUtilization: 85 type: Utilization - type: Resource resource: name: memory target: averageUtilization: 85 type: Utilization Modify HPA instances as needed. After HPA instances are created, manually edit them to better optimize your configuration based on your workload requirements. Disable HPA if necessary. To remove HPA for a component, remove the hpa field or set it to false: backend: workerSpec: hpa: false Additional notes Enabling HPA overrides and ignores any replicas values set in the APIManager CR for apicast-production , backend-listener , and backend-worker . For vertical scaling, set resource requests equal to limits. Since the HPA scales based on a default utilization of 85%, setting aside additional resources for limits is unnecessary. 1.7. Installing 3scale API Management with the operator using Oracle as the system database As a Red Hat 3scale API Management administrator, you can install the 3scale with the operator using the Oracle Database. By default, 3scale 2.15 has a component called system that stores configuration data in a MySQL database. You can override the default database and store your information in an external Oracle Database. Note The Oracle Database is not supported with OpenShift Container Platform (OCP) versions 4.2 and 4.3 when you are performing an operator-only installation of 3scale. For more information, refer to the Red Hat 3scale API Management Supported Configurations page. In this documentation myregistry.example.com is used as an example of the registry URL. Replace it with your registry URL. Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content. Prerequisites A container registry to push container images, accessible by the OCP cluster where 3scale installed. An installation of the 3scale operator . Do not install the APIManager CR, as it will be created in the following procedure. A supported version of the Oracle Database accessible from your OpenShift cluster. Access to the Oracle Database SYSTEM user for installation procedures. To install 3scale with the operator using Oracle as the system database, use the following steps: Preparing the Oracle Database Building a custom system container image Installing 3scale API Management with Oracle using the operator 1.7.1. Preparing the Oracle Database As a 3scale administrator, you must fully prepare the Oracle Database for your 3scale installation when you decide to use it for the System component. Procedure Create a new database. Apply the following settings: ALTER SYSTEM SET max_string_size=extended SCOPE=SPFILE; Configure a database user There are two options for setting up Oracle Database integration in 3scale: with or without providing the Oracle SYSTEM user password. 3scale uses the SYSTEM user only for the initial setup, which consist in creating a regular user and granting it the required privileges. The following SQL commands will set up a regular user with proper permissions. ( {DB_USER} and {DB_PASSWORD} are placeholders that need to be replaced with actual values): CREATE USER {DB_USER} IDENTIFIED BY {DB_PASSWORD}; GRANT unlimited tablespace TO {DB_USER}; GRANT create session TO {DB_USER}; GRANT create table TO {DB_USER}; GRANT create view TO {DB_USER}; GRANT create sequence TO {DB_USER}; GRANT create trigger TO {DB_USER}; GRANT create procedure TO {DB_USER}; Using the SYSTEM user: Provide the SYSTEM user password in ORACLE_SYSTEM_PASSWORD field of the system-database secret. The regular user does not need to exist before the installation. It will be created by the 3scale initialization script. Provide the desired username and password for the regular user in the connection string, for example, oracle-enhanced://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME} in the URL field of the system-database secret. The password for the regular Oracle Database non-system user must be unique and not match the SYSTEM user password. If the user with the specified username already exists, the 3scale initialization script will attempt to update the password using the following command: ALTER USER {DB_USER} IDENTIFIED BY {DB_PASSWORD} Your database configuration might prevent this command from completing successfully if the parameters PASSWORD_REUSE_TIME and PASSWORD_REUSE_MAX are set in a way that restricts reusing the same password. Manual setup of the regular database user: You do not need to provide the ORACLE_SYSTEM_PASSWORD in the system-database secret. The regular database user (not SYSTEM ) specified in the connection string in the URL field of the system-database secret needs to exist prior to the 3scale installation. The regular user used for the installation must have all the privileges listed above. Additional resources For information on creating a new database, see the Oracle Database 19c documentation. 1.7.2. Building a custom system container image Procedure Download 3scale OpenShift templates from the GitHub repository and extract the archive: From the Instant Client Downloads page, download: A client: It can be either basic-lite or basic . The ODBC driver . The SDK for Oracle Database 19c. For 3scale, use Instant Client Downloads for Linux x86-64 (64-bit) For ppc64le and 3scale, use Oracle Instant Client Downloads for Linux on Power Little Endian (64-bit) Check the table for the following Oracle software component versions: Oracle Instant Client Package: Basic or Basic Light Oracle Instant Client Package: SDK Oracle Instant Client Package: ODBC Table 1.6. Oracle 19c example packages for 3scale Oracle 19c package name Compressed file name Basic instantclient-basic-linux.x64-19.18.0.0.0dbru.zip Basic Light instantclient-basiclite-linux.x64-19.18.0.0.0dbru.zip SDK instantclient-sdk-linux.x64-19.18.0.0.0dbru.zip ODBC instantclient-odbc-linux.x64-19.18.0.0.0dbru.zip Table 1.7. Oracle 19c example packages for ppc64le and 3scale Oracle 19c package name Compressed file name Basic instantclient-basic-linux.leppc64.c64-19.3.0.0.0dbru.zip Basic Light instantclient-basiclite-linux.leppc64.c64-19.3.0.0.0dbru.zip SDK instantclient-sdk-linux.leppc64.c64-19.3.0.0.0dbru.zip ODBC instantclient-odbc-linux.leppc64.c64-19.3.0.0.0dbru.zip Note If the client packages versions downloaded and stored locally do not match with the ones 3scale expects, 3scale will automatically download and use the appropriate ones in the following steps. Place your Oracle Database Instant Client Package files into the system-oracle-3scale-2.15.0-GA/oracle-client-files directory. Build the custom system Oracle-based image. The image tag must be a fixed image tag as in the following example: Push the system Oracle-based image to a container registry accessible by the OCP cluster. This container registry is where your 3scale solution is going to be installed: 1.7.3. Installing 3scale API Management with Oracle using the operator Procedure Set up the Oracle Database URL connection string and Oracle Database system password by creating the system-database secret with the corresponding fields. See, External databases installation for the Oracle Database. Install your 3scale solution by creating an APIManager CR. Follow the instructions in Deploying 3scale API Management using the operator . The APIManager CR must specify the .spec.system.image field set to the system's Oracle-based image you built: 1.8. Troubleshooting common 3scale API Management installation issues This section contains a list of common installation issues and provides guidance for their resolution. deployment leaving dirty persistent volume claims Incorrectly pulling from the Docker registry Permission issues for MySQL when persistent volumes are mounted locally Unable to upload logo or images Test calls not working on OpenShift APIcast on a different project from 3scale failing to deploy 1.8.1. deployment leaving dirty persistent volume claims Problem A deployment attempt leaves a dirty Persistent Volume Claim (PVC) causing the MySQL container to fail to start. Cause Deleting a project in OpenShift does not clean the PVCs associated with it. Solution Procedure Find the PVC containing the erroneous MySQL data with the oc get pvc command: # oc get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES AGE backend-redis-storage Bound vol003 100Gi RWO,RWX 4d mysql-storage Bound vol006 100Gi RWO,RWX 4d system-redis-storage Bound vol008 100Gi RWO,RWX 4d system-storage Bound vol004 100Gi RWO,RWX 4d Stop the deployment of the system-mysql pod by clicking cancel deployment in the OpenShift Container Platform (OCP) console. Delete everything under the MySQL path to clean the volume. Start a new system-mysql deployment. 1.8.2. Incorrectly pulling from the Docker registry Problem The following error occurs during installation: svc/system-redis - 1EX.AMP.LE.IP:6379 deployment/system-redis deploys docker.io/rhscl/redis-32-rhel7:3.2-5.3 deployment #1 failed 13 minutes ago: config change Cause OpenShift searches for and pulls container images by issuing the docker command. This command refers to the docker.io Docker registry instead of the registry.redhat.io Red Hat Ecosystem Catalog. This occurs when the system contains an unexpected version of the Docker containerized environment. Solution Procedure Use the appropriate version of the Docker containerized environment. 1.8.3. Permission issues for MySQL when persistent volumes are mounted locally Problem The system-msql pod crashes and does not deploy causing other systems dependant on it to fail deployment. The pod log displays the following error: [ERROR] Cannot start server : on unix socket: Permission denied [ERROR] Do you already have another mysqld server running on socket: /var/lib/mysql/mysql.sock ? [ERROR] Aborting Cause The MySQL process is started with inappropriate user permissions. Solution Procedure The directories used for the persistent volumes MUST have the write permissions for the root group. Having read-write permissions for the root user is not enough as the MySQL service runs as a different user in the root group. Execute the following command as the root user: Execute the following command to prevent SElinux from blocking access: 1.8.4. Unable to upload logo or images Problem Unable to upload a logo - system-app logs display the following error: Cause Persistent volumes are not writable by OpenShift. Solution Procedure Ensure your persistent volume is writable by OpenShift. It should be owned by root group and be group writable. 1.8.5. Test calls not working on OpenShift Problem Test calls do not work after creation of a new service and routes on OpenShift. Direct calls via curl also fail, stating: service not available . Cause 3scale requires HTTPS routes by default, and OpenShift routes are not secured. Solution Procedure Ensure the secure route checkbox is clicked in your OpenShift router settings. 1.8.6. APIcast on a different project from 3scale API Management failing to deploy Problem APIcast deploy fails (pod does not turn blue). You see the following error in the logs: You see the following error in the pod: Cause The secret was not properly set up. Solution Procedure When creating a secret with APIcast v3, specify apicast-configuration-url-secret : 1.9. Additional resources External Components Specification System database

Commands:
oc new-project 3scale-project
Removing the operator will not remove any of its custom resource definitions or managed resources. If your operator has deployed applications on the cluster or configured off-cluster resources, these will continue to run and need to be cleaned up manually. This action removes the operator as well as the Operator deployments and pods, if any. Any operands and resources managed by the operator, including CRDs and CRs, are not removed. The web console enables dashboards and navigation items for some operators. To remove these after uninstalling the operator, you might need to manually delete the operator CRDs.
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager-sample spec: wildcardDomain: example.com
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager-sample spec: system: appSpec: replicas: 1 sidekiqSpec: replicas: 1 zync: appSpec: replicas: 1 queSpec: replicas: 1 backend: cronSpec: replicas: 1 listenerSpec: replicas: 1 workerSpec: replicas: 1 apicast: productionSpec: replicas: 1 stagingSpec: replicas: 1 wildcardDomain: example.com
xdg-open https://3scale-admin.3scale-project.example.com
oc get secret system-seed -n <namespace> -o json | jq -r .data.ADMIN_USER | base64 -d oc get secret system-seed -n <namespace> -o json | jq -r .data.ADMIN_PASSWORD | base64 -d
oc get secret system-seed -n <namespace> -o json | jq -r .data.MASTER_USER | base64 -d oc get secret system-seed -n <namespace> -o json | jq -r .data.MASTER_PASSWORD | base64 -d
externalComponents: backend: redis: true system: database: true redis: true zync: database: true
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: allProxy: http://forward-proxy:80 stagingSpec: allProxy: http://forward-proxy:81
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: httpProxy: http://forward-proxy:80 stagingSpec: httpProxy: http://forward-proxy:81
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: httpsProxy: https://forward-proxy:80 stagingSpec: httpsProxy: https://forward-proxy:81
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: noProxy: theStore,company.com,big.red.com stagingSpec: noProxy: foo,bar.com,.extra.dot.com
local cjson = require('cjson') local PolicyChain = require('apicast.policy_chain') local policy_chain = context.policy_chain local logging_policy_config = cjson.decode([[ { "enable_access_logs": false, "custom_logging": "\"{{request}}\" to service {{service.id}} and {{service.name}}" } ]]) policy_chain:insert( PolicyChain.load_policy('logging', 'builtin', logging_policy_config), 1) return { policy_chain = policy_chain, port = { metrics = 9421 }, }
oc create secret generic custom-env-1 --from-file=./env1.lua
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager-apicast-custom-environment spec: wildcardDomain: <desired-domain> apicast: productionSpec: customEnvironments: - secretRef: name: custom-env-1 stagingSpec: customEnvironments: - secretRef: name: custom-env-1
oc apply -f apimanager.yaml
oc create secret generic my-first-custom-policy-secret --from-file=./apicast-policy.json --from-file=./init.lua --from-file=./my-first-custom-policy.lua
oc label secret my-first-custom-policy-secret apimanager.apps.3scale.net/watched-by=apimanager
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager-apicast-custom-policy spec: apicast: stagingSpec: customPolicies: - name: my-first-custom-policy version: "0.1" secretRef: name: my-first-custom-policy-secret - name: my-second-custom-policy version: "0.1" secretRef: name: my-second-custom-policy-secret productionSpec: customPolicies: - name: my-first-custom-policy version: "0.1" secretRef: name: my-first-custom-policy-secret - name: my-second-custom-policy version: "0.1" secretRef: name: my-second-custom-policy-secret
oc apply -f apimanager.yaml
apiVersion: v1 kind: Secret metadata: name: myjaeger stringData: config: |- { "service_name": "apicast", "disabled": false, "sampler": { "type": "const", "param": 1 }, "reporter": { "queueSize": 100, "bufferFlushInterval": 10, "logSpans": false, "localAgentHostPort": "jaeger-all-in-one-inmemory-agent:6831" }, "headers": { "jaegerDebugHeader": "debug-id", "jaegerBaggageHeader": "baggage", "TraceContextHeaderName": "uber-trace-id", "traceBaggageHeaderPrefix": "testctx-" }, "baggage_restrictions": { "denyBaggageOnInitializationFailure": false, "hostPort": "127.0.0.1:5778", "refreshInterval": 60 } } type: Opaque
oc create -f myjaeger.yaml
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: apimanager1 spec: apicast: stagingSpec: openTracing: enabled: true tracingLibrary: jaeger tracingConfigSecretRef: name: myjaeger productionSpec: openTracing: enabled: true tracingLibrary: jaeger tracingConfigSecretRef: name: myjaeger
oc apply -f apimanager1.yaml
oc create secret tls mycertsecret --cert=server.crt --key=server.key
spec: apicast: productionSpec: httpsPort: 8443 httpsVerifyDepth: 1 httpsCertificateSecretRef: name: mycertsecret
spec: apicast: stagingSpec: httpsPort: 8443 httpsVerifyDepth: 1 httpsCertificateSecretRef: name: mycertsecret
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: lvh.me resourceRequirementsEnabled: false
apiVersion: v1 kind: Secret metadata: name: backend-redis stringData: REDIS_STORAGE_URL: "redis://backend-redis-storage" REDIS_STORAGE_SENTINEL_HOSTS: "redis://sentinel-0.example.com:26379,redis://sentinel-1.example.com:26379, redis://sentinel-2.example.com:26379" REDIS_STORAGE_SENTINEL_ROLE: "master" REDIS_QUEUES_URL: "redis://backend-redis-queues" REDIS_QUEUES_SENTINEL_HOSTS: "redis://sentinel-0.example.com:26379,redis://sentinel-1.example.com:26379, redis://sentinel-2.example.com:26379" REDIS_QUEUES_SENTINEL_ROLE: "master" type: Opaque
apiVersion: v1 kind: Secret metadata: name: system-redis stringData: URL: "redis://system-redis" SENTINEL_HOSTS: "redis://sentinel-0.example.com:26379,redis://sentinel-1.example.com:26379, redis://sentinel-2.example.com:26379" SENTINEL_ROLE: "master" type: Opaque
apiVersion: v1 kind: Secret metadata: name: system-database stringData: URL: "mysql2://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}" type: Opaque
[mysqld] default_authentication_plugin=mysql_native_password
apiVersion: v1 kind: Secret metadata: name: system-database stringData: URL: "postgresql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}" type: Opaque
apiVersion: v1 kind: Secret metadata: name: system-database stringData: URL: "oracle-enhanced://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}" ORACLE_SYSTEM_PASSWORD: "{SYSTEM_PASSWORD}" type: Opaque
apiVersion: v1 kind: Secret metadata: name: zync stringData: DATABASE_URL: postgresql://<zync-db-user>:<zync-db-password>@<zync-db-host>:<zync-db-port>/zync_production ZYNC_DATABASE_PASSWORD: <zync-db-password> type: Opaque
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: lvh.me externalComponents: system: database: true
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: backend: listenerSpec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/hostname" operator: In values: - ip-10-96-1-105 - key: "beta.kubernetes.io/arch" operator: In values: - amd64 system: memcachedTolerations: - key: key1 value: value1 operator: Equal effect: NoSchedule - key: key2 value: value2 operator: Equal effect: NoSchedule
affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: deploymentConfig: apicast-production topologyKey: kubernetes.io/hostname - weight: 99 podAffinityTerm: labelSelector: matchLabels: deploymentConfig: apicast-production topologyKey: topology.kubernetes.io/zone
affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: deploymentConfig: apicast-production topologyKey: kubernetes.io/hostname - weight: 99 podAffinityTerm: labelSelector: matchLabels: deploymentConfig: apicast-production topologyKey: topology.kubernetes.io/zone
spec: apicast: stagingSpec: replicas: 0 productionSpec: replicas: 0 backend: listenerSpec: replicas: 0 workerSpec: replicas: 0 cronSpec: replicas: 0 zync: appSpec: replicas: 0 queSpec: replicas: 0 system: appSpec: replicas: 0 sidekiqSpec: replicas: 0
bundle exec rake zync:resync:domains
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::<target_bucket_name>", 1 "arn:aws:s3:::<target_bucket_name>/*" 2 ] } ] }
[ { "AllowedMethods": [ "GET" ], "AllowedOrigins": [ "http://*" ] } ]
apiVersion: v1 kind: Secret metadata: creationTimestamp: null name: aws-auth stringData: AWS_ACCESS_KEY_ID: <ID_123456> AWS_SECRET_ACCESS_KEY: <ID_98765544> AWS_BUCKET: <mybucket.example.com> AWS_REGION: eu-west-1 type: Opaque
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: <example_apimanager> namespace: <3scale_test> spec: wildcardDomain: lvh.me system: fileStorage: simpleStorageService: configurationSecretRef: name: aws-auth
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: <apimanager_sample> namespace: <3scale_test> spec: wildcardDomain: lvh.me system: fileStorage: simpleStorageService: configurationSecretRef: name: s3-credentials sts: enabled: true audience: openshift
kind: Secret apiVersion: v1 metadata: name: s3-credentials namespace: 3scale data: AWS_ROLE_ARN: arn:aws:iam::ID:role/ROLE AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/openshift/serviceaccount/token AWS_BUCKET: <mybucket.example.com> AWS_REGION: eu-west-1 type: Opaque
apiVersion: v1 kind: Pod metadata: name: system-sidekiq-1-zncrz namespace: 3scale-test spec: containers: . volumeMounts: - mountPath: /var/run/secrets/openshift/serviceaccount name: s3-credentials readOnly: true . volumes: - name: s3-credentials projected: defaultMode: 420 sources: - serviceAccountToken: audience: openshift expirationSeconds: 3600 path: token
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: lvh.me system: database: postgresql: {}
oc login -u <user> <url> oc project <3scale-project>
oc patch secret system-smtp -p '{"stringData":{"address":"<your_address>"}}' oc patch secret system-smtp -p '{"stringData":{"port":"587"}}' oc patch secret system-smtp -p '{"stringData":{"username":"<your_username>"}}' oc patch secret system-smtp -p '{"stringData":{"password":"<your_password>"}}' oc patch secret system-smtp -p '{"stringData":{"authentication":"plain"}}' oc patch secret system-smtp -p '{"stringData":{"openssl.verify.mode":"none"}}'
oc rollout restart deployment/system-app oc rollout restart deployment/system-sidekiq
oc rollout status deployment/system-app oc rollout status deployment/system-sidekiq
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: backend: listenerSpec: resources: requests: memory: "150Mi" cpu: "300m" limits: memory: "500Mi" cpu: "1000m" system: appSpec: developerContainerResources: limits: cpu: 1500m memory: 1400Mi requests: cpu: 150m memory: 600Mi masterContainerResources: limits: cpu: 1500m memory: 1400Mi requests: cpu: 150m memory: 600Mi providerContainerResources: limits: cpu: 1500m memory: 1400Mi requests: cpu: 150m memory: 600Mi zync: databaseResources: requests: memory: "111Mi" cpu: "222m" limits: memory: "333Mi" cpu: "444m"
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: api.vmogilev01.0nno.s1.devshift.org resourceRequirementsEnabled: false apicast: stagingSpec: priorityClassName: openshift-user-critical productionSpec: priorityClassName: openshift-user-critical backend: listenerSpec: priorityClassName: openshift-user-critical cronSpec: priorityClassName: openshift-user-critical workerSpec: priorityClassName: openshift-user-critical redisPriorityClassName: openshift-user-critical system: appSpec: priorityClassName: openshift-user-critical sidekiqSpec: priorityClassName: openshift-user-critical searchdSpec: priorityClassName: openshift-user-critical searchdSpec: priorityClassName: openshift-user-critical memcachedPriorityClassName: openshift-user-critical redisPriorityClassName: openshift-user-critical database: postgresql: priorityClassName: openshift-user-critical zync: appSpec: priorityClassName: openshift-user-critical queSpec: priorityClassName: openshift-user-critical databasePriorityClassName: openshift-user-critical
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: example.com resourceRequirementsEnabled: false backend: listenerSpec: labels: backendLabel1: sample-label1 backendLabel2: sample-label2 apicast: stagingSpec: labels: apicastStagingLabel1: sample-label1 apicastStagingLabel2: sample-label2
apiVersion: capabilities.3scale.net/v1beta1 kind: OpenAPI metadata: name: ownertest namespace: threescale annotations: "insecure_skip_verify": "true" spec: openapiRef: secretRef: name: myopenapi namespace: threescale productSystemName: testProduct
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: example.com apicast: stagingSpec: annotations: anno-sample1: anno1 backend: listenerSpec: annotations: anno-sample2: anno2
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: resourceRequirementsEnabled: true/false
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: backend: listenerSpec: replicas: X workerSpec: replicas: Y cronSpec: replicas: Z
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: apicast: productionSpec: replicas: X stagingSpec: replicas: Z
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: system: appSpec: replicas: X sidekiqSpec: replicas: Z
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: zync: appSpec: replicas: X queSpec: replicas: Z
apicast: productionSpec: serviceCacheSize: 20 stagingSpec: serviceCacheSize: 10
oc get dc/apicast-staging -o yaml
oc get dc/apicast-production -o yaml
apicast-staging - name: APICAST_SERVICE_CACHE_SIZE value: '10'
apicast-production - name: APICAST_SERVICE_CACHE_SIZE value: '20'
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: wildcardDomain: example.com resourceRequirementsEnabled: true apicast: productionSpec: hpa: true backend: listenerSpec: hpa: true workerSpec: hpa: true
apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: backend-worker spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: backend-worker minReplicas: 1 maxReplicas: 5 metrics: - type: Resource resource: name: cpu target: averageUtilization: 85 type: Utilization - type: Resource resource: name: memory target: averageUtilization: 85 type: Utilization
backend: workerSpec: hpa: false
ALTER SYSTEM SET max_string_size=extended SCOPE=SPFILE;
CREATE USER {DB_USER} IDENTIFIED BY {DB_PASSWORD}; GRANT unlimited tablespace TO {DB_USER}; GRANT create session TO {DB_USER}; GRANT create table TO {DB_USER}; GRANT create view TO {DB_USER}; GRANT create sequence TO {DB_USER}; GRANT create trigger TO {DB_USER}; GRANT create procedure TO {DB_USER};
ALTER USER {DB_USER} IDENTIFIED BY {DB_PASSWORD}
curl -L -o system-oracle-3scale-2.15.0-GA.tar.gz https://github.com/3scale/system-oracle/archive/refs/tags/3scale-2.15.0-GA.tar.gz tar -xzf system-oracle-3scale-2.15.0-GA.tar.gz
cd system-oracle-3scale-2.15.0-GA docker build . --tag myregistry.example.com/system-oracle:2.15.0-1
docker push myregistry.example.com/system-oracle:2.15.0-1
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: example-apimanager spec: imagePullSecrets: - name: threescale-registry-auth - name: custom-registry-auth system: image: "myregistry.example.com/system-oracle:2.15.0-1" externalComponents: system: database: true
oc get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES AGE backend-redis-storage Bound vol003 100Gi RWO,RWX 4d mysql-storage Bound vol006 100Gi RWO,RWX 4d system-redis-storage Bound vol008 100Gi RWO,RWX 4d system-storage Bound vol004 100Gi RWO,RWX 4d
svc/system-redis - 1EX.AMP.LE.IP:6379 deployment/system-redis deploys docker.io/rhscl/redis-32-rhel7:3.2-5.3 deployment #1 failed 13 minutes ago: config change
[ERROR] Cannot start server : on unix socket: Permission denied [ERROR] Do you already have another mysqld server running on socket: /var/lib/mysql/mysql.sock ? [ERROR] Aborting
chmod -R g+w /path/for/pvs
chcon -Rt svirt_sandbox_file_t /path/for/pvs
Errno::EACCES (Permission denied @ dir_s_mkdir - /opt/system/public//system/provider-name/2
update acceptor rejected apicast-3: pods for deployment "apicast-3" took longer than 600 seconds to become ready
Error synching pod, skipping: failed to "StartContainer" for "apicast" with RunContainerError: "GenerateRunContainerOptions: secrets \"apicast-configuration-url-secret\" not found"
oc create secret generic apicast-configuration-url-secret --from-literal=password=https://<ACCESS_TOKEN>@<TENANT_NAME>-admin.<WILDCARD_DOMAIN>