Isolation of concurrent read and write transactions on the same file

The disclosure provides for isolation of concurrent read and write transactions on the same file, thereby enabling higher file system throughput relative to serial-only transactions. Race conditions and lock contentions in multi-writer scenarios are avoided in file stat (metadata) updates by the use of an aggregator to merge updates of committed transactions to maintain file stat truth, and an upgrade lock that enforces atomicity of file stat access, even while still permitting multiple processes to concurrently read from and/or write to the file data. The disclosure is applicable to generic file systems, whether native or virtualized, and may be used, for example, to speed access to database files that require prolonged input/output (I/O) transaction time periods.

BACKGROUND

Some file systems use transactions to provide atomicity, consistency, isolation, and durability (ACID) properties for file operations, and in some use cases, concurrent read/write transactions on the same file may be frequent. Previously, concurrency has been achieved by using a byte-level range lock, which allows read/write operations that do not have overlapping ranges to be executed concurrently. However, access to the file data may result in changes to the file's metadata. For example, multiple processes reading disjointed sections of the file data could result in concurrent attempts to update the metadata, specifically access time (atime).

Unfortunately, metadata changes cannot be isolated using a range lock, resulting in a race condition. Additionally, updating metadata prior to a transaction commit introduces potential inaccuracy for other processes. If a first process changes the file metadata, for example change time (ctime) and file size, prior to committing a write transaction, a second process reads the changed metadata, and then the first process fails prior to the transaction commit, then the second process will be operating on corrupted (e.g., untruthful) data.

SUMMARY

An exemplary system for isolating concurrent read and write transactions on a file comprises: a processor; a computer-readable medium storing instructions that are operative when executed by the processor to: obtain an upgrade lock of file stat data for the file; copy at least a portion of the file stat into a private storage; for each mergeable transaction in a transaction group, merge an update of the current mergeable transaction into the file stat portion in the private storage; atomically store the file stat portion of the private storage into the file stat data for the file; and release the upgrade lock of the file stat data for the file.

DETAILED DESCRIPTION

FIG. 1illustrates a race condition problem in a multi-writer scenario100. A file102is being written to concurrently by two separate processes: process1and process2. Process1writes to a first portion104of file102, and process2writes to a second portion106of file102. Both process1and process2increase the size of file102and add new blocks. During this time, but prior to completion by either process1or process2, a third process, process3, is attempting to read from file102. Upon process1and process2completing their respective writing operations, process1and process2each attempts to access metadata108to update data such as a timestamp, the file size, and the number of blocks. Metadata108is a single shared object, and therefore, a race condition110exists, in which one of process1and process2will write its update first, followed by the other. A problem with this scheme is that each of process1and process2bases its own update on the prior version of metadata108, with the second one over-writing (rather than incorporating) the other's update. Thus, the final version of metadata108reflects only the updates from one of process1or process2, rather than both.

To overcome such a problem associated with multi-writer scenario100ofFIG. 1and other problems, various aspects of the systems and methods described herein provide for isolation of concurrent read and write transactions on the same file, thereby enabling higher file system throughput relative to serial-only transactions. Race conditions and lock contentions in multi-writer scenarios are avoided in file stat (metadata) updates by the use of an aggregator to merge updates of committed transactions to maintain file stat truth, and an upgrade lock that enforces atomicity of file stat access, even while still permitting multiple processes to concurrently read from and/or write to the file data. The disclosure is applicable to generic file systems, whether native or virtualized, and may be used, for example, to speed access to database files that require prolonged input/output (I/O) transaction time periods. Thus, the current disclosure may be advantageously employed for large files, when I/O requires a long time period, and serialized transactions (with each employing exclusive access) would otherwise introduce significant delays.

In this manner, the disclosure is able to maintain truth of metadata, such as timestamps, file size, and the number of blocks that the file occupies while enabling quicker reads and writes. Some examples may be implemented on a virtual distributed file system (VDFS) in which file data is changed concurrently, while the metadata is updated atomically. In order to provide both isolation and concurrency, a Read-Copy-Update (RCU) stat data structure and shared pointer are used to store metadata updates, with the possibility of maintaining multiple version of the metadata for a single file. A read transaction reads from the latest version of metadata when it begins, and that metadata version (accessed by the read transaction) does not change, even if a newer version of the metadata is created.

Each transaction holds its own metadata update privately prior to the transaction commit. Upon the commit, the transaction appends a new, public version of the metadata. Atomic pointer load and store operations are used to avoid race conditions, and obsolete metadata is deleted after the final read transaction commits. Isolation of the updates is ensured because uncommitted metadata changes are private. Concurrency of read and write operations is unaffected, because there is no blocking operation in the RCU process. A group commit operation is used to aggregate metadata changes from multiple transactions, thereby merging updates to timestamps, file size, and the number of blocks.

It should be understood that any of the examples herein are non-limiting. As such, the present disclosure is not limited to any particular embodiments, aspects, concepts, structures, functionalities or examples described herein. Rather, the present disclosure may be used in various ways that provide benefits and advantages in computing systems including virtualized computing environments.

FIG. 2illustrates an example of shared metadata in a shared cache200. Shared cache200has a cache entry202that includes a key204and a file stat206. Key204may be an inode number or another unique file identifier. File stat206is metadata for a file, such as for example metadata108for file102(shown inFIG. 1). File stat206includes stat data208(which may be a pointer to the data or may be the actual data itself). Stat data208does not include data that is in RCU stat214(which is the type of data that may change with a read or write operation), but is instead additional metadata that does not change as a result of a read or write operation. Examples of stat data208include file attributes, such as a file name, owner, and permissions. An RCU stat pointer212is a shared pointer that indicates the memory location of RCU stat214. Because RCU stat pointer212is shared, it does not create any race conditions.

The illustrated example of RCU stat214includes ctime, atime, mtime, file size, and number of blocks, which are mergeable data. Modification time (mtime) describes when the content of the file most recently changed. Some file systems do not compare data written to a file with the prior copy, so if a process overwrites part of a file with the same data as had previously existed in that location, mtime is updated even though the contents did not actually change. Access time (atime) identifies when the file was most recently opened for reading. Because some computer configurations are faster at reading data than at writing it, updating atimes after every read operation can become expensive. Therefore, some computer configurations mitigate this cost by storing atimes at a coarser granularity than mtimes and ctimes, so that a file which is read repeatedly in a short time frame only needs its atime updated once. Change time (ctime) in a UNIX brand operating system reflects time when certain file metadata, rather than file contents, were last changed, such as file permissions or file owner. For a WINDOWS brand operating system, ctime means creation time.

File stat206further includes a reference count210that counts the number of processes currently sharing RCU stat pointer212. When a new process, either read or write, accesses the file (for which file stat206holds the metadata), the new process obtains shared RCU stat pointer212and reference count210increments. When that process releases shared RCU stat pointer212, reference count210decrements. Additional detail is illustrated inFIG. 5.

FIG. 3illustrates a flowchart300of example operations for isolating concurrent read and write transactions on a file. The example operations illustrated by flowchart300are used with file stat206ofFIG. 2when reading from or writing to a file, and may be implemented while another process is writing to the same file in another transaction and yet another process is reading from the same file in yet another transaction. Operation302includes obtaining, by the writing process, a shared lock of the file stat data, for example, file stat206ofFIG. 2. This prevents the file from being deleted. Operation304includes obtaining, by the writing process, a shared pointer, which points to at least a portion of the file stat, such as shared RCU stat pointer212. For example, shared RCU stat pointer212points to a RCU stat214, which is at least a portion of the data contained by file stat206. The write process then stores a private copy of RCU stat214in operation306, which is used for generating mergeable update information in operation310.

Operation308includes file read and write operations (R/W operations or I/O operations), such as, for example, writing, by the writing process, to the file in a transaction. Operation310includes storing a mergeable update for the transaction, which reflects the necessary metadata update information to the private copy of RCU stat214, which was saved during operation306. The mergeable update is stored separately from other mergeable updates stored by other processes. This prevents the mergeable update information from one process from being overwritten by another process. Mergeable update information includes timestamps, file size, and number of blocks. Operation312includes releasing, by the writing process, the shared pointer of the at least a portion of the file stat (e.g., shared RCU stat pointer212), and operation314includes committing the transaction. Operation316then includes releasing, by the writing process, the shared lock of the file stat. Any unused copies of the RCU stat pointer212(that pointed to obsolete copies of RCU stat214) are deleted in operation318, freeing up the memory that had been occupied by the obsolete copies of RCU stat214. Additional detail on this operation is described in relation toFIG. 5B.

The status of the reference count for the shared pointer is also illustrated. In operation302, when the writing process obtains a shared lock of the file stat, the reference count (e.g., reference count210of file stat206) shows a value of 1, indicating that another process is already accessing the file. In operation304, when the writing process obtains the shared pointer, the reference count increments to a value of 2. In operation312, when the writing process releases the shared pointer, the reference count decrements to a value of 1. Additional detail on this operation is provided in the description ofFIG. 5A. It should be understood that, although the operation312is described for a writing process, other examples of operation312involve a read processes. It should also be understood that multiple concurrent file operations, each operating according to flowchart300, are possible.

FIG. 4illustrates a flowchart400of example operations for isolating concurrent read and write transactions on a file. The example operations illustrated by flowchart400are used with file stat206ofFIG. 2when merging multiple concurrent metadata updates from multiple I/O processes, and is performed by an aggregator (such as, for example, aggregator814ofFIG. 8). For example, each of multiple processes may have concurrently performed the operations of flowchart300, and now the updates from those transactions are to be merged. The set of transaction updates to be merged together during the operations of flowchart400is a transaction group.

Operation402includes obtaining an upgrade lock of file stat data for the file (e.g. file stat206for file102). The upgrade lock enforces atomic operations for the merging the updates, and avoiding race conditions. Operation404includes copying at least a portion of the file stat into a private storage. In some examples, copying at least a portion of the file stat into the private storage comprises copying RCU stat data including timestamp, file size, and a number of blocks for the file into the private storage. Looping operation406controls cycling through operation408for each mergeable transaction update in a transaction group, and exiting into operation410when complete. Operation408includes merging an update of the current mergeable transaction (the transaction which is being addressed by the current iteration of operation40) into the file stat portion (e.g., RCU stat) in the private storage. In some examples, merging an update comprises selecting the maximum timestamp value for at least one timestamp selected from the list consisting of atime, ctime, and mtime. In some examples, merging an update comprises selecting a maximum file size value as a final file size value for the file. In some examples, merging an update comprises adding a delta of a number of blocks to an initial number of blocks to determine a final number of blocks. Additional detail on this operation is provided in the description ofFIG. 6.

Operation412includes atomically storing the file stat portion of the private storage into the file stat data for the file. In some examples, this includes storing RCU stat pointer212for the merged RCU stat214into file stat206. In some examples, atomically storing the file stat portion of the private storage into the file stat data for the file comprises creating a new pointer for the file stat portion, such as for example, creating a new RCU stat pointer212. Additional detail on this operation is provided in the description ofFIG. 5B. Atomicity of storing the file stat portion of the private storage into the file stat data for the file is provided by the upgrade lock. Operation414then releases the upgrade lock of the file stat data for the file (e.g., releases the upgrade lock of file stat206).

FIG. 5Aillustrates an example change scheme500afor a pointer count value, such as reference count210, when accessing file stat206ofFIG. 2. At stage502, a read/write process (R/W process) attaches to a file to perform an R/W operation, and reference count210(for RCU stat pointer212) increments to 2, as indicated by status box504a. At stage506, the process releases the file, for example, by committing a transaction, and reference count210decrements back down to 1, as indicated by status box504b. This describes the operations performed on reference count210during operations304and312ofFIG. 3. In scheme500a, the release of the file preserves the RCU stat pointer, because reference count210still has a value of 1.

FIG. 5B, however, illustrates an alternative example change scheme500bin which a new RCU stat pointer212is created for a write process. Scheme500bmay also be used when accessing file stat206ofFIG. 2. In operation512a reading process attaches to the file, for example operating according to flowchart300. As indicated by status box514a, reference count210ahas a value of 1. In operation516, a merging operation, such as according to flowchart400, begins. Rather than reference count210aincrementing to a value of 2, reference count210aremains at 1, and a new pointer is created, with a reference count210bhaving a value of 1. The reference count210bindicates that the merging operation is ongoing. This is shown in status box514b(with status unchanged from status box514a) and in a new status box518a.

When the read process releases its RCU stat pointer, in operation520, reference count210adecrements to zero, as indicated in status box514c. This results in the deletion of the first RCU pointer, according to operation318(ofFIG. 3). Status box518b(with status unchanged from status box514a) indicates that the new RCU stat pointer is unperturbed. The new RCU stat pointer then becomes the RCU stat pointer for the file, and points to the merged updated RCU stat. Therefore any new processes accessing the file have access to the correct metadata, and the prior read process was not interrupted.

FIG. 6illustrates an example mergeable update rule set600for updating file stat206ofFIG. 2. Rule set600is used, for example, during operation408of flowchart400. Rule set includes rules602,604, and606, which further includes rules608,610, and612. Rule602applies to timestamps and file size values. Rule602is to select the maximum value among various values of the mergeable updates and the initial RCU stat data. The illustrated example indicates that an initial file size was 100; writer A changes the file size from 100 to 110, which is a delta of +10; Writer B changes the file size from 100 to 120, which is a delta of +20. Rule602uses the absolute values of the file sizes, rather than the delta (e.g., difference) values, and results in the selection of 120 as the file size. A similar result occurs for the timestamps (atime, ctime, and mtime). The maximum timestamp values indicate the latest times.

Rule604is to sum all the delta values for the number of blocks, from each of the updates, and add the sum to the initial value to produce the final value of the number of blocks. The illustrated example indicates an initial numblocks (number of blocks) value of 50. Writer A writes 10 blocks, so the numblocks delta is +10, and the absolute value of numblocks is 60. Writer B then writes concurrently with Writer C. Writer B writes 20 blocks, so the numblocks delta is +10, and the absolute value of numblocks is 70, determined by adding 10 to the absolute value of 60 (from Writer A). Writer C writes 1 block, so the numblocks delta is +1, and the absolute value of numblocks is 61, determined by adding 1 to the absolute value of 60 (from Writer A). Rule604uses the delta values, adding 50 to 10, plus 10, plus 1, to calculate 71. This is the final value for the number of blocks.

Rule606results in atomicity for the upgrade lock, even while permitting concurrent R/W operations. Rule608, which is a part of rule606, permits a new upgrade lock if there is no prior upgrade lock. The new upgrade lock can exist with multiple shared locks, which permits the concurrent R/W operations. Rule610, which is also a part of rule606, denies a new upgrade lock if there is currently a prior-existing upgrade lock still in force, thereby enforcing the atomic operations. An upgrade lock can co-exist with shared locks. Rule612, which is also a part of rule606, denies a new upgrade lock if there is currently a prior-existing exclusive lock still in force. An upgrade lock cannot exist with an exclusive lock. Exclusive lock is used for operations that cannot happen concurrently with other operations. For example, deleting a file requires the exclusive lock on it.

FIG. 7illustrates a flowchart700showing a method for isolating concurrent read and write transactions on a file. Flowchart700incorporates aspects ofFIGS. 3-6and some examples are performed by computing device802ofFIG. 8. Process1is set to write to a file in operation702, process2is also set to write to the same file (concurrently with process1) in operation712, and process3is set to concurrently read from the same file in operation722. Process1initializes operations according to flowchart300, in operation704, while process2initializes a parallel set of operations according to flowchart300, in operation714. Concurrently, process3initializes yet another parallel set of operations according to flowchart300, in operation724. When operations704and714complete (each a manifestation of the operations of flowchart300), operation706initiates the operations of flowchart400. Because processes1and2were write operations, when process3completes the operations of flowchart300, the original RCU stat pointer is deleted (see operation318ofFIG. 3and alsoFIG. 5B).

FIG. 8illustrates a block diagram of an example computing architecture800, including an example computing device802(a computer system), that implements aspects disclosed herein. Example computing architecture800, for example, implements the operations of flowchart700ofFIG. 7. Computing device802has at least a processor804and a memory area806that holds program code and data808. Memory area806is any device allowing information, such as computer executable instructions and/or other data, to be stored and retrieved. For example, memory area806may include one or more random access memory (RAM) modules, persist memory, phase change memory, flash memory modules, hard disks, shingled disks, solid-state disks, and/or optical disks. Program code808comprises computer executable instructions and associated data, including a virtual machine (VM) platform810, and a VDFS812.

An aggregator814performs the operations of flowchart400ofFIG. 4, with shared cache200(ofFIG. 2) and file102(ofFIG. 1), using mergeable update rule set600(ofFIG. 6). Process1, process2, and process3represent any of the R/W processes (I/O processes) described herein, such as the processes ofFIGS. 1, 5A, and 5B. Private copy of RCU stat816and private copy of RCU stat818represent private copies of RCU stat214that were created by various processes during operation306flowchart300(ofFIG. 3). Mergeable update information820and mergeable update information822represent the mergeable updates for various transaction that were written during operation310of flowchart300(ofFIG. 3) and merged together in operation408of flowchart400(ofFIG. 4).

Aggregator private copy824is the copy of the portion of the file stat (e.g., RCU stat) that was placed into private storage during operation404and then copied into file stat data for the file during operation412(both ofFIG. 4). New pointer826is the pointer whose creation was depicted inFIG. 5B, and which corresponds with reference count210b.

Other logic and storage828includes any other applications, data, and storage used during the operations of computing device802. An input/output (I/O) module830permits storage of program code and data808in a storage location832, and accepting inputs form users. I/O module830also permits communication over network834with a remote node836, which may be another manifestation of computing device802. Computing device802represent any device executing instructions (e.g., as application programs, operating system functionality, or both) to implement the operations and functionality described herein. Computing device802may include any portable or non-portable device including a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, portable media player, desktop personal computer, kiosk, and/or tabletop device. Additionally, computing device802may represent a group of processing units or other computing devices, such as in a cloud computing system or service. Processor804may include any quantity of processing units and may be programmed to execute any components of program code808comprising computer executable instructions for implementing aspects of the disclosure. In some embodiments, processor804is programmed to execute instructions such as those illustrated in the figures.

ADDITIONAL EXAMPLES

An example system for isolating concurrent read and write transactions on a file comprises: a processor; a computer-readable medium storing instructions that are operative when executed by the processor to: obtain an upgrade lock of file stat data for the file; copy at least a portion of the file stat into a private storage; for each mergeable transaction in a transaction group, merge an update of the current mergeable transaction into the file stat portion in the private storage; atomically store the file stat portion of the private storage into the file stat data for the file; and release the upgrade lock of the file stat data for the file.

An example method of isolating concurrent read and write transactions on a file comprises: obtaining an upgrade lock of file stat data for the file; copying at least a portion of the file stat into a private storage; for each mergeable transaction in a transaction group, merging an update of the current mergeable transaction into the file stat portion in the private storage; atomically storing the file stat portion of the private storage into the file stat data for the file; and releasing the upgrade lock of the file stat data for the file.

One or more exemplary non-transitory computer storage medium having computer-executable instructions that, upon execution by a processor, cause the processor to at least perform operations that comprise: obtaining an upgrade lock of file stat data for the file; copying at least a portion of the file stat into a private storage; for each mergeable transaction in a transaction group, merging an update of the current mergeable transaction into the file stat portion in the private storage; atomically storing the file stat portion of the private storage into the file stat data for the file; and releasing the upgrade lock of the file stat data for the file.

Alternatively, or in addition to the other examples described herein, examples include any combination of the following:copying data including timestamp, file size, and a number of blocks for the file into the private storage;selecting the maximum timestamp value for at least one timestamp selected from the list consisting of: atime, ctime, and mtime;selecting a maximum file size value as a final file size value for the file;adding a delta of a number of blocks to an initial number of blocks to determine a final number of blocks;atomicity of storing the file stat portion of the private storage into the file stat data for the file is provided by the upgrade lock;atomically storing the file stat portion of the private storage into the file stat data for the file comprises creating a new pointer for the file stat portion;while a first writing process is writing to the file in a first transaction, obtaining, by a second writing process, a shared lock of the file stat data; obtaining, by the second writing process, a shared pointer of at least a portion of the file stat; writing, by the second writing process, to the file in a second transaction; storing a mergeable update for the second transaction; releasing, by the second writing process, the shared pointer of the at least a portion of the file stat; committing the second transaction; and releasing, by the second writing process, the shared lock of the file stat data; andwhile the second writing process is writing to the file, reading from the file with a reading process.
Exemplary Operating Environment

The operations described herein may be performed by a computer or computing device. The computing devices comprise processors and computer readable media. By way of example and not limitation, computer readable media comprise computer storage media and communication media. Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media are tangible, non-transitory, and are mutually exclusive to communication media. In some examples, computer storage media are implemented in hardware. Exemplary computer storage media include hard disks, flash memory drives, NVMe drives, digital versatile discs (DVDs), compact discs (CDs), floppy disks, tape cassettes, and other solid-state memory. In contrast, communication media typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery media.

Although described in connection with an exemplary computing system environment, examples of the disclosure are operative with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with aspects of the disclosure include, but are not limited to, mobile computing devices, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices.

Aspects of the disclosure transform a general-purpose computer into a special purpose computing device when programmed to execute the instructions described herein.

While some virtualized embodiments are described with reference to VMs for clarity of description, the disclosure is operable with other forms of virtual computing instances (VCIs). A VCI may be a VM, a container, and/or any other type of virtualized computing instance.

In examples that involve a hardware abstraction layer on top of a host computer (e.g., server), the hardware abstraction layer allows multiple containers to share the hardware resource. These containers, isolated from each other, have at least a user application running therein. The hardware abstraction layer thus provides benefits of resource isolation and allocation among the containers. In some examples, VMs may be used alternatively or in addition to the containers, and hypervisors may be used for the hardware abstraction layer. In these examples, each VM generally includes a guest operating system in which at least one application runs.

For the container examples, it should be noted that the disclosure applies to any form of container, such as containers not including a guest operating system, referred to herein as “OS-less containers” (see, e.g., www.docker.com). OS-less containers implement operating system-level virtualization, wherein an abstraction layer is provided on top of the kernel of an operating system on a host computer. The abstraction layer supports multiple OS-less containers each including an application and its dependencies. Each OS-less container runs as an isolated process in user space on the host operating system and shares the kernel with other containers. The OS-less container relies on the kernel's functionality to make use of resource isolation (CPU, memory, block I/O, network, etc.) and separate namespaces and to completely isolate the application's view of the operating environments. By using OS-less containers, resources may be isolated, services restricted, and processes provisioned to have a private view of the operating system with their own process ID space, file system structure, and network interfaces. Multiple containers may share the same kernel, but each container may be constrained to only use a defined amount of resources such as CPU, memory and I/O.

The detailed description provided above in connection with the appended drawings is intended as a description of a number of embodiments and is not intended to represent the only forms in which the embodiments may be constructed, implemented, or utilized. Although these embodiments may be described and illustrated herein as being implemented in devices such as a server, computing devices, or the like, this is only an exemplary implementation and not a limitation. As those skilled in the art will appreciate, the present embodiments are suitable for application in a variety of different types of computing devices, for example, PCs, servers, laptop computers, tablet computers, etc.

The term “computing device” and the like are used herein to refer to any device with processing capability such that it can execute instructions. Those skilled in the art will realize that such processing capabilities are incorporated into many different devices and therefore the terms “computer”, “server”, and “computing device” each may include PCs, servers, laptop computers, mobile telephones (including smart phones), tablet computers, and many other devices. Any range or device value given herein may be extended or altered without losing the effect sought, as will be apparent to the skilled person. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

While no personally identifiable information is tracked by aspects of the disclosure, examples have been described with reference to data monitored and/or collected from the users. In some examples, notice may be provided to the users of the collection of the data (e.g., via a dialog box or preference setting) and users are given the opportunity to give or deny consent for the monitoring and/or collection. The consent may take the form of opt-in consent or opt-out consent.

It will be understood that the benefits and advantages described above may relate to one embodiment or may relate to several embodiments. When introducing elements of aspects of the disclosure or the examples thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. The term “exemplary” is intended to mean “an example of.”