SUSE Labs Conference 2019 - A Whirlwind Tour Around Our Public Cloud Efforts
Channel: SUSE Labs
Date: 2019-09-17

None

SUSE has a presence in the Public Cloud since 2010, with the first image released in Amazon Elastic Compute Cloud (EC2) on October 4, 2010. Since this time many things have changed. One significant event was on September 25, 2014, when with the event of Shellshock we did not only need to refresh all the released active images at the time, but a new update infrastructure had to be pressed into service as well. The existing infrastructure at the time encountered various issues and there was no salvaging it in a reasonable amount of time to address the Shellshock concerns and making the updates available. Luckily the new infrastructure had been under design and development for a while and was in the final stages in testing. In 2014 we only supported EC2 and Azure and the update infrastructure in both frameworks was different generally increasing the maintenance.  After Shellshock this problem was resolved.

Fast forward to 2019. The update infrastructure design and implementation that was pressed into service on that late/early September day has grown to over 220 servers managed by SUSE Manager. The team has grown from 2 to 12 and with the latest round of image refreshes due to SACK we pushed 69 unique images out the door. The update infrastructure operates in almost 70 unique regions around the globe, and the team maintains over 200 packages.

Statistics are interesting but, where are the problems and whats going on?

We are almost ready to release all our images in a mostly automated fashion using mash https://github.com/SUSE-Enceladus/mash.


Mash

Not just a step in the beer making process, it is a tool that will automate our image release process. Mash is designed as a pipeline that is comprised of various services. At present we have mash running through the testing stage in Amazon and Azure. The pipeline when completed will be composed of the following steps:

- Download an image from IBS
- Upload the image to a given CSP
- Test the image (img-prrof)
- Replicate the image if needed
- Release the image

The remaining manual step for the time being will be to update our Public-cloud INformation Tracker (pint) data. This is targetd to be automated eventually.

Each service is independent and additional services can be inserted into the pipeline if needed. Services communicate over a message bus. Once we are confident things works as we want them to we will open the pipeline to other teams. Users out side of the public cloud team will be restricted to run the pipeline only through the testing service. The goal is to have mash in use by the Product and Maintenance QA teams and test as many updates as possible in the Public Cloud.


img-proof

Another original development by the Public Cloud Team, https://github.com/SUSE-Enceladus/img-proof is a tetsing application to test our images in the various frameworks. This project ws started before mash, with the integration into mash in mind. The img-prrof native tests are comprised of a number of smke tests, reboot, verify the proper license is in the image, check the motd, etc. to ensure basic correctness of the image. Thanks to the work by the SLE Product QA team img-proof is also integrated into openQA and is used during the product development cycle. The Product QA team runs additional tests, kernel, networking etc. in the instances.


The Update Infrastructure

The previously mentioned update infrastructure consists of so called region servers and earlier this year was transitioned from SMT to RMT without significant customer interruption. Customer impact was limited to SLES 11 running instances but we were able to resolve this in reasonably short order.

SLES and SLES For SAP on-demand (PAYG) instances are resold by the Public CLoud providers and entitle the user to access of our updates. The connection to the update servers is fully automated and when the use logs in via ssh 'zypper up' will pull the latest and greatest.

Future work is to allow users of our BYOS (Bring Your Own Subscription) images to also hook into our update infrastructure.


Feature enablement

Things in the Public Cloud move a bit faster. Partially because it's mostly software and partially because we are not told about new whizbang HW until very late in the cycle. In Azure work mostly revolves around new instance type support and new features made available by Hyper-V to the Linux guest. ALmost of all these changes require kernel work. In Amazon we have the ENA and EFA drivers that receive the most changes. In Google we've had relatively few requests that effect the kernel. For the most part when a request reaches us the code is in flight to upstream or has already been accepted upstream. Working with each provider is a bit different.

Robert Schweikert

https://labsconf.suse.cz/labs2019/talk/RSUVCG