Patent Description:
A new version of a software application may change a persistency model associated with data instances managed by the application. Accordingly, updating to the new version requires updates to the corresponding persistent data. Conventionally, such updates to persistent data were performed during a downtime period in which the application to be updated is taken offline and is therefore unavailable to users. Depending on the amount of persistent data, the update may require a considerable amount of time, which may impact the enterprise negatively.

Recent techniques execute an update to an application while the application running (i.e., serving incoming user requests). These techniques create clones of the tables which are changed by the update and replicate data from the tables to the clones. Once the replicated data is verified, incoming user requests are switched to the updated application.

After the switch, the incoming requests will typically cause updates to the persistency. If an error in the update is then detected, it is possible to restore the persistency to a state prior to the switch and to re-direct incoming requests to the prior version of the application. The restoration of the database causes a loss of the data updates which occurred after the switch. In an alternative, the persistency is not restored and the incoming requests are simply directed to the prior version of the application. Since the prior version would be unaware of certain persistency changes (e.g., added columns or added tables), it would therefore be unable to read any data written thereto, also resulting in data "loss".

This data loss (i.e., either restoring the database to an earlier point in time or effectively disabling access to new columns or tables) may adversely affect data consistency within the software application and within the mesh of services and systems in conjunction with which the application operates. For example, if the application includes fields which require unique numbers for legal or other reasons, simply re-entering data will lead to new numbers for the same records. If data was created in other systems based on the lost data, the created data is no longer consistent. Even if the lost data is created anew, the corresponding foreign-key-relations may not be restored.

Systems are desired which facilitate reversion from an updated version of an application to a prior version of the application while reducing the likelihood of data loss. Such systems may also provide for evaluating and patching the updated version prior to switchover to the updated version. "<NPL>" describes QuantumDB, which is a tool-supported approach that abstracts the evolution process away from the web service without locking tables. This approach allows to redeploy a web service without needing to take it offline even when a database schema change is necessary. In addition, QuantumDB puts no restrictions on the method of deployment, supports schema changes to multiple tables using changesets, and does not subvert foreign key constraints during the evolution process. <CIT> describes implementations that include providing, by a deploy tool, clone data components in the first database system, each clone data component being a copy of a data component, defining, by the deploy tool, a source-side green access schema in the first database system, the green access schema providing views to the clone data components, and providing, by a replication system and based on statements received from the deploy tool, consumer-side clone data components in the first database system, each consumer-side clone data component being a copy of a respective data component. The implementation further include defining, by a replication system and based on statements received from the deploy tool, a consumer-side green access schema in the first database system, the green access schema providing views to the source-side clone data components, and, during execution of the upgrade, replicating, by a handler of the replication system, data from at least one source-side data component to a consumer-side component. <CIT> describes initiating an upgrade of a first version of a database application to a second version of a database application that both have a same data schema. The first database application has a first access schema such that at least one table in the data schema is linked to the first access schema. The second version of the database application has a second access schema such that at least one table in the data schema is linked to the second access schema. The first access schema differs from the second access schema. Subsequently, concurrent access is provided for each access schema to at least one database table in the data schema to both the first version of the database application and the second version of the database application. The concurrent access is enabled by using separate read and write channels. Related apparatus, systems, techniques and articles are also described.

The underlying technical problem is solved by the subject-matter having the features of the independent claims. Additional embodiments are defined in the dependent claims. The following description is provided to enable any person in the art to make and use the described embodiments. Various modifications, however, will remain readily-apparent to those in the art.

Some embodiments provide a system to rollback an update to a software application after users have commenced productive use of the updated application, such that data created during the productive use of the updated application is usable by the prior version of the application after the rollback. For example, some errors may become evident only after evaluating the read and write performance of the updated application. Embodiments allow initiation of a rollback as described above after determination of such an error. The erroneous update may then be patched and the above process may repeat - with users using the patched version to create new data and, if needed, rolling back to the prior version again without data loss.

<FIG> illustrate update of a current application to an updated application according to some embodiments. The illustrated components may be implemented using any suitable combinations of computing hardware and/or software that is or becomes known. Such combinations may include cloud-based implementations in which computing resources are virtualized and allocated elastically. In some embodiments, two or more components are implemented by a single computing device.

Persistency <NUM> may comprise a database system as is known in the art. Persistency <NUM> may comprise a single node or distributed database system, and may be implemented using hardware such as but not limited to an on-premise computer server or a cloud-based virtual machine. Persistency <NUM> may be managed by program code of query-responsive database management system (DMBS). According to some embodiments, persistency <NUM> is an "in-memory" database, in which all data is loaded into memory upon startup and requested changes are made thereto. The changed data is flushed from time-to-time to persistency <NUM> to move older data versions from memory to persistent storage or to persist a database snapshot.

Persistency <NUM> stores database tables within data schema <NUM> and views on the database tables within access schema <NUM>. Access schema <NUM> may also store program code of database procedures and other artifacts as in known in the art. The database tables of data schema <NUM> conform to a logical data model in a relational data schema <NUM> associated with current application <NUM>.

Current application <NUM> comprises processor-executable program code executed within runtime container <NUM>. Runtime container <NUM> and the execution environment may be provided by a server as is known in the art. The server may comprise, for example, a web server, an application server, a proxy server, a network server, and/or a server pool. The server accepts requests for application services and provides such services to any number of client devices.

Current application <NUM> and persistency <NUM> may comprise a production system which is deployed by an enterprise to service incoming user requests. During operation, users <NUM> request functionality, dispatcher <NUM> directs corresponding requests to current application <NUM> and, in turn, current application <NUM> issues corresponding queries to persistency <NUM>. The queries may include read write statements which result in data being stored to, deleted from, and/or edited within persistency <NUM>.

As shown in <FIG>, current application <NUM> is separated from data schema <NUM> by a first access schema <NUM>. Access schema <NUM> contains a projection view for every database table in data schema <NUM>, and application <NUM> queries the database tables via their corresponding views. A view in the access schema is an updatable projection view into a database table. Views are stackable in that one view may be a view of another one or more view.

In one example, a table T1 of data schema <NUM> includes fields F1-F5, and a projection view Table of access schema <NUM> defines fields F1-F4 of table T1. Consequently, a query received from current application <NUM> to retrieve records of view Table will result in retrieval of data from fields F1-F4 (and not F5) of table T1. Multiple views may define different sets of fields of a same table (e.g., [F1-F3], F2-F5], (F1, F3, F5]), and a single view may define fields of two or more database tables.

Multiple independent access schemas can be provided for a same data schema. For example, a first access schema may support communication between a first application and a first data model in a data schema, and a second access schema may support communication between a second application and a second data model in the same data schema. Such multiple access schemas may be used to support an update as described herein procedure.

<FIG> illustrates the beginning of an update process according to some embodiments. The update process is intended to update current application <NUM> with updated application <NUM>. It will be assumed that updated application <NUM> is associated with one or more changes to the data model in data schema <NUM>. Accordingly, the update process begins by extending the data model in data schema <NUM> based on the changes defined by updated application <NUM>. Examples of changes include adding a column to a database table, changing a type of a column, and/or cloning a table to accept new records generated by the updated application. After extension, schema <NUM> includes tables <NUM> accessible via access schema <NUM> and tables <NUM> which include a subset of tables <NUM>.

A second access schema <NUM> is also prepared in persistency <NUM>. The second access schema <NUM> contains projection view referencing tables <NUM> of data schema <NUM>, and is intended for use by updated application <NUM> to access tables <NUM>. In this regard, updated application <NUM> is also deployed to runtime container <NUM> to prepare updated application <NUM> for execution.

Next, data from tables <NUM> is replicated to those tables <NUM> which are not in common with tables <NUM>, according to the defined data model of tables <NUM>. Such replication may include transformation of data from a column of a table <NUM> to another column of a table <NUM>. As shown in <FIG>, current application <NUM> continues to receive and service incoming requests prior to and during the replication. Accordingly, database triggers are set to ensure that any new data which should be replicated to the not-common tables <NUM> is replicated in order to keep the non-common tables up-to-date.

Once the replication is complete (except for ongoing database trigger-initiated replications), tables <NUM> may be tested for consistency using "smoke tests" or other read operation-based data consistency tests that are or become known. Testing may include executing a test instance of updated application <NUM> within runtime container <NUM> to issue read statements to access schema <NUM>. If the tests are not successful, updated application <NUM> and/or access schema <NUM> may be modified to correct any errors and similar testing may be run again. If errors continue, application <NUM>, schema <NUM> and not-common tables <NUM> may simply be discarded, allowing productive operation to continue using current application <NUM> as shown in <FIG>.

If the consistency tests are successful, dispatcher <NUM> is instructed to direct the incoming requests to updated application <NUM> as shown in <FIG>. Additionally, database triggers are activated to reverse the direction of the above-mentioned replication. As will be described in detail below, the reversal results in new data written to tables <NUM> during operation of updated application <NUM> to be replicated to corresponding ones of tables <NUM>. Consequently, tables <NUM> remain up-to-date despite the lack of queries issued by current application <NUM> while incoming requests are being served by updated application <NUM>.

Updating tables <NUM> during operation of updated application <NUM> as described herein allows an efficient rollback to current application <NUM>. For example, if operation of updated application <NUM> as shown in <FIG> is determined to be unsatisfactory (e.g., due to errors or performance issues), dispatcher <NUM> may be instructed to direct the incoming requests back to application <NUM> as shown in <FIG>. Application <NUM> may then continue operation using the data generated during operation of application <NUM>. As described above, the process may return from <FIG> after patching of application <NUM> and/or access schema <NUM>, or to the configuration shown in <FIG>.

Once satisfactory operation of updated application <NUM> is verified, application <NUM>, schema <NUM> and tables <NUM> which are not within tables <NUM> may be discarded. Productive operation thereby continues using updated application <NUM>, schema <NUM> and tables <NUM> as shown in <FIG>.

<FIG> illustrate update of current application <NUM> to updated application <NUM> according to some embodiments. It is assumed that current application <NUM> of <FIG> is executing within runtime container <NUM> based on incoming requests from users (not shown) during productive operation. Persistency <NUM> stores data schema <NUM> and access schema <NUM> associated with current application <NUM>. Data schema <NUM> includes table TABX, in addition to other unshown tables. Similarly, access schema <NUM> includes view TABX on table TABX, in addition to other unshown views on other tables (and possibly other views on one or more columns of table TABX).

Updated application <NUM> requires the presence of column C within table TABX and accesses table TABX using view TABX which includes a column pointing to column C of table TABX. As mentioned above, an update process may include extension of an existing data model to satisfy the data model of an updated application. Accordingly, as shown in <FIG>, the data model in data schema <NUM> is extended to add a column C to table TABX. Since executing current application <NUM> is unaware of column C of table TABX, column C does not include data. As also shown in <FIG>, new access schema <NUM> is prepared within persistency <NUM> and updated application <NUM> is deployed to runtime container <NUM> configured to access the database via access schema <NUM>. The dashed lines of <FIG> are intended to indicate that updated application <NUM> and access schema <NUM> are not being used to serve incoming user requests during productive operation.

<FIG> illustrates table TABX after continued execution of application <NUM>. As shown, additional rows have been added to table TABX by virtue of interactions between application <NUM> and access schema <NUM>. The additional rows to not include values for column C.

It will be assumed that any migration of other table data (as will be described below) completes after the scenario shown in <FIG>. Moreover, it will also be assumed that any desired consistency checks of the migrated data are passed, resulting in redirection of incoming requests to updated application <NUM> executing within runtime container <NUM>. Such requests may cause application <NUM> to interact with view TABX of access schema <NUM> which, as shown in <FIG>, results in the addition of two rows to table TABX of data schema <NUM>. Since view TABX in access schema <NUM> includes new column C of table TABX, the added records include data for column C.

It will now be assumed that reversion back to application <NUM> is desired due to an error or other reason. <FIG> illustrates, by means of dashed and solid lines, the redirection of incoming requests to current application <NUM>, resulting in use of current application <NUM> and access schema <NUM> to serve the incoming requests. During such operation, current application <NUM> may access the records of table TABX which were added during operation of updated application <NUM>. However, current application <NUM> may only access columns K, A and B of the added records, because only those columns are referenced by view TABX of access schema <NUM>.

<FIG> illustrate update of current application <NUM> to updated application <NUM> according to some embodiments. Current application <NUM> of <FIG> is executing within runtime container <NUM> based on incoming requests from users (not shown) during productive operation. Persistency <NUM> stores data schema <NUM> and access schema <NUM> associated with current application <NUM>. Data schema <NUM> includes table TABY, in addition to other unshown tables, and access schema <NUM> includes view TABY on table TABY, in addition to other unshown views on other tables.

Updated application <NUM> changes a type of column B (e.g., to B#). In the example, B# is <NUM> of the value of B. Embodiments may include any conversion or transformation between data types. <FIG> illustrates extension of the data model in data schema <NUM> to satisfy the data model of updated application <NUM>. As shown, the data model in data schema <NUM> is extended to add a column B# to table TABY.

<FIG> also shows preparation of new access schema <NUM> within persistency <NUM> and deployment of updated application <NUM> to runtime container <NUM>. As described above, update of an application includes migration of existing data to corresponding columns and/or tables of an updated data model. Accordingly, column B# of table TABY is populated during the migration with appropriately-transformed values of its related column B. Moreover, database trigger <NUM> is activated to operate as will be described below.

<FIG> illustrates data schema <NUM> after continued execution of application <NUM>, which has caused the addition of a row to table TABY. Since view TABY of access schema <NUM> does not reference column B# of table TABY, no value is included in column B# of the new record. However, trigger <NUM> operates to transform the value of B from any newly-added record to a corresponding value of B# and to store the corresponding value of B# in the record, as shown in <FIG>. Trigger <NUM> operates in this manner in a case that incoming requests are being received and served by current application <NUM>, as is the case in <FIG>.

It is then assumed that the migration of data is completed and any consistency checks are satisfied, resulting in redirection of incoming requests to updated application <NUM> executing within runtime container <NUM> as shown in <FIG>. Due to the operation of trigger <NUM>, updated application <NUM> may access the records of table TABY which were added by application <NUM> before and after extension of the data model in data schema <NUM>.

As shown in <FIG>, the incoming requests may cause application <NUM> to interact with view TABY of access schema <NUM>. In the present example, this interaction results in the addition of a row to table TABY of data schema <NUM>. Since access schema <NUM> points to columns K, A and B# of table TABY, the added record does not include a data value for column B.

In a case that incoming requests are being received and served by updated application <NUM> (as shown in <FIG>), trigger <NUM> operates to transform the value of B# from any newly-added record to a corresponding value of B and to store the corresponding value of B in the record. <FIG> illustrates such transformation and storage (i.e., of value <NUM>) according to some embodiments.

In a case that reversion back to application <NUM> is desired due to an error or other reason, incoming requests may simply be redirected to current application <NUM> as shown in <FIG>. Advantageously, current application <NUM> may access the records of table TABY which were added during operation of updated application <NUM>. Current application <NUM> may only access columns K, A and B of the added records, since those columns and not column B# are referenced by view TABY of access schema <NUM>. In addition, trigger <NUM> may continue to operate to populate column B# of newly-added records of table TABY so that such records may be accessed by updated application <NUM> and access schema <NUM> (or patched versions thereof) if it is determined to attempt the update again.

<FIG> illustrate update of a current application to an updated application which adds new records to a table during the update process according to some embodiments. <FIG> illustrate update of current application <NUM> to updated application <NUM> according to some embodiments. Updated application <NUM> is intended to add new records to table TABZ during the update process.

<FIG> illustrates extension of the data model in data schema <NUM> based on the data model required by updated application <NUM>. In particular, data schema <NUM> is extended to add a "clone" table TABZ#. Access schema <NUM> is also prepared within persistency <NUM> and updated application <NUM> is deployed to runtime container <NUM>. Database triggers <NUM> and <NUM> are respectively associated with each of tables TABZ and TABZ# and configured to operate as will be described below.

The data of table TABZ is migrated to table TABZ# during the above-mentioned initial data migration such that the data stored therein is identical, as shown in <FIG>. Continued execution of application <NUM> may result in a record being added to table TABZ, as shown in <FIG>. In response to addition of the record to table TABZ, database trigger <NUM> causes replication of the record in table TABZ#, as shown in <FIG>.

It is then assumed that incoming requests are redirected to updated application <NUM> executing within runtime container <NUM> as shown in <FIG> also shows the addition of a record to table TABZ# due to operation of updated application <NUM>. In response to addition of the record to table TABZ#, database trigger <NUM> operates to replicate the added record in table TABZ, as shown in <FIG>. According to some embodiments, and to avoid entering an endless replication loop, trigger <NUM> replicates data to table TABZ only if incoming requests are being received and served by updated application <NUM> and trigger <NUM> replicates data to table TABZ# only if incoming requests are being received and served by current application <NUM>.

<FIG> illustrates reversion back to current application <NUM>. As shown, current application <NUM> may access the records of table TABZ which were replicated thereto during operation of updated application <NUM>. Moreover, trigger <NUM> may continue to operate to replicate records to table TABZ in a case that it is decided to revert back to updated application <NUM> and access schema <NUM> or to patched versions of updated application <NUM> and access schema <NUM>.

<FIG> illustrates process <NUM> to update a current application to an updated application according to some embodiments. Process <NUM> and the other processes described herein may be performed using any suitable combination of hardware and software. Software program code embodying these processes may be stored by any non-transitory tangible medium, including a fixed disk, a volatile or non-volatile random access memory, a DVD, a Flash drive, or a magnetic tape, and executed by any number of processing units, including but not limited to processors, processor cores, and processor threads. Such processors, processor cores, and processor threads may be implemented by a virtual machine provisioned in a cloud-based architecture. Embodiments are not limited to the examples described below.

Initially, at S505, a first application is executed to receive incoming user requests. The first application is associated with a first access schema and a first model in a data schema of a database system. The first application, first access schema and first data model in a data schema may comprise a production system which is deployed by an enterprise to service incoming user requests as described above with respect to <FIG>, for example. Execution of the first application to receive the user requests results in write statements which update data stored in the data schema.

<FIG> is a diagram illustrating the states of current and updated applications, access schemas and data schemas during various stages of process <NUM> according to some embodiments. For example, State <NUM> represents execution (i.e., "in-use") of a first application in conjunction with a first access schema and a first data model in a data schema of a database system.

An updated application is determined at S510. The updated application may comprise an updated version of the first application, and is associated with a second access schema and a second data model for the data schema. At S515, the first data model in the data schema is extended based on the second data model. As described above, examples of such extension include but are not limited to adding a column to a database table, changing a type of a column, and/or cloning a table to accept new records generated by the updated application.

Next, at S520, the second access schema is prepared in the database system. The second access schema is prepared to allow the updated application to access the database tables of the extended data model in the data schema via projection views for each table in the data schema. State <NUM> of <FIG> represents the state of the update process after S520.

The updated application is deployed to a runtime container at S525. At S530, data of the first data model in the data schema is migrated to the second data model and database triggers are activated to replicate any new changes to the first data model to the second data model. The migration/replication may include transformation of data from a column of a table to another column of the table and/or migrating data to a cloned table as described above. S525 and S530 are represented by State <NUM> of <FIG>.

A consistency test is performed on the migrated data at S535. A consistency test, for example, may include executing a test instance of the updated application to issue read statements to the second access schema and to analyze the result sets for consistency. Testing at S535 is represented by State <NUM>, which illustrates that migration/replication may continue during the testing since the first application continues to service incoming requests and causing corresponding updates to the data of the data schema.

If it is determined at S540 that the test is not successful, the system may return to State <NUM>. Specifically, the updated application and/or second access schema may be modified (i.e., patched) at S545 to correct any errors, with flow returning to S520 to prepare the patched second access schema and to S525 to deploy the patched updated application for testing thereof at S535.

If the test is successful, the incoming requests are switched to the updated application as represented by State <NUM>. At State <NUM>, the updated application, second access schema and second data model in the data schema are operated to serve the incoming user requests. As shown, the first application remains executing, although in a stand-by mode because it is not receiving user requests. Moreover, database triggers are activated to reverse the replication direction as described in the above examples. As described, such reversal ensures that the first data model remains up-to-date while incoming requests are being served by the updated application.

It will be assumed that an error is detected at S560, while the system resides at State <NUM>. If so, the system is returned to State <NUM> by switching the incoming user requests back to the first application and by again reversing the direction of the replication at S565. At S570, it may be determined to continue with or abort the update process. If it is determined to abort the process, the updated application is stopped and the database triggers are deactivated at S575 to move the system to State <NUM>. The updated application, the second access schema and the second data model in the data schema may then be discarded to return the system to State <NUM>.

If it is determined at S570 to continue the update process, flow may return to S545 (i.e., State <NUM>) and continue as described above. Alternatively, flow may return to State <NUM> to again attempt execution of the updated application.

In a case that no error is detected at S560, it may be determined to continue execution of the updated application. Accordingly, at S580, the first application is stopped and the database triggers are deactivated, as represented. Next, the first application, the first access schema and the first data model in the data schema may be discarded to move the system to State <NUM> and complete the update process. With reference to the example of <FIG>, S580 may include dropping column B from data schema <NUM> and maintaining column B#. Referring to the example of <FIG>, S580 may include dropping table TABZ from data schema <NUM> while maintaining table TABZ#.

According to the invention, the updated application may include a change or new feature which creates or uses data in a manner which is semantically different that the first application. Such "irreversible" changes or features may be toggled off during execution of the updated application at State <NUM>, and toggled on once the state of the system advances to State <NUM>. By toggling the features off at State <NUM>, it remains possible to transition back to State <NUM> in which the first application executes. Moreover, the features may be toggled on at State <NUM> for testing, and toggled off prior to State <NUM> depending on the results of testing.

According to an example of the foregoing, it is assumed that a database system includes a table storing data and access control information. An updated application is available which adds a column to the table specifying whether or not the data of each record is only visible to a small audience of people with elevated access rights. The column may be named "top-secret" and newly-created records which are "top-secret" include "yes" in the column while other records include "no".

If the updated application is executed at State <NUM> to create records with the new column, and a rollback to State <NUM> is then executed, the new column is not visible to the first application and is not evaluated thereby. Accordingly, any records created by the updated application which include "yes" in the new column may be visible to all users having regular access to the table. The creation of "top-secret" records should therefore be toggled off at State <NUM>, and only toggled on once the state transitions to State <NUM> because rollback from State <NUM> to State <NUM> is not possible.

If a problem associated with the feature occurs during execution of the updated application at State <NUM>, the feature may be toggled off and all records in which "top-secret" is "yes" may be marked as unreadable by any user, independent of access rights. This approach prevents access to the records by any user until the feature is fixed, and allows access to the records by appropriate users once the feature is fixed and toggled on.

<FIG> illustrates a cloud-based database deployment according to some embodiments. The illustrated components may comprise cloud-based compute resources residing in one or more public clouds providing self-service and immediate provisioning, autoscaling, security, compliance and identity management features.

User device <NUM> may interact with applications executing on application server <NUM>, for example via a Web Browser executing on user device <NUM>, in order to create, read, update and delete data managed by database system <NUM>. Administrator system <NUM> may control a process to update an application executing on application server <NUM> to a new version as described herein. Such a control may require communication with database system <NUM> to prepare an access schema and extend a data model in a data schema based on the new version of the application as described herein.

Two or more of such computing devices may be located remote from one another and may communicate with one another via any known manner of network(s) and/or a dedicated connection. Each component or device may comprise any number of hardware and/or software elements suitable to provide the functions described herein as well as any other functions. For example, any computing device used in an implementation of a system according to some embodiments may include a processor to execute program code such that the computing device operates as described herein.

Claim 1:
A system comprising:
a memory storing processor-executable program code; and
at least one processing unit to execute the processor-executable program code to cause the system to:
execute (S505) a first application (<NUM>) to receive incoming user requests, the first application (<NUM>) associated with a first access schema and a first data model in a data schema of a database system;
while the first application (<NUM>) is receiving incoming user requests:
extend (S515) the first data model in the data schema based on a second data model of a second application (<NUM>);
prepare (S520) a second access schema of the second application (<NUM>) in the database system;
migrate (S530) data of the first data model in the data schema to the second data model;
execute a first one or more database triggers (<NUM>) to replicate data of the first data model in the data schema to the second data model while the first application (<NUM>) is receiving incoming user requests;
execute the second application (<NUM>);
re-direct the incoming user requests to be received by the executing second application (<NUM>); and
stop the replication of data of the first data model in the data schema to the second data model; and
while the second application (<NUM>) is receiving incoming user requests:
execute a second one or more database triggers (<NUM>) to replicate data of the second data model in the data schema to the first data model;
wherein the at least one processing unit further is to execute the processor-executable program code to cause the system to:
delete the first access schema and the first data model in the data schema from the database system,
after deletion of the first access schema and the first data model in the data schema from the database system:
toggle on a feature of the second application (<NUM>);
execute the second application (<NUM>) to generate records while the feature is toggled on; and
toggle off the feature and mark the records as unreadable.