Find appropriate AWS policy I have created a user group in AWS ("developers"). Now I want to attach a policy to the group which will grant the users permission to list, create and update Amplify apps. I am struggling to determine the appropriate policy.

I see that the "AdministratorAccess" policy includes access to the AWS Amplify service, but I don't want the developers to have administrator level access. What are my options?

Edit:
When the specific user accesses the Amplify console the following error is displayed

List apps call failed User: arn:aws:iam::XXXXXXXXXX:user/myUser is not authorized to perform: amplify:ListApps on resource: arn:aws:amplify:eu-west-1:XXXXXXXXXX:apps/*

If you want to create your own IAM policy, you can use the visual editor present in IAM. Using visual editor, you can construct a new policy from scratch using the very intuitive GUI.

Goto IAM, then policies, then use visual editor.

This link detail how we can create role for providing access to applications in aws and you can grant these role to your group .

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

Another way you can create your own policy and assign to group .

https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html


Change container size in a template from the Marketplace I am trying to use a template from the Azure Marketplace. Everything works, but it defaults to storage sizes that are by orders of magnitude too large for my usecase.

At the end of the process, I can "kind of" export the template files in text form - at  least parameters and variables, but there are no values for the storage here. I assume that those are hard coded inside whatever scripts they use internally to deploy that application. I see no way to have a look at or modify those scripts.

Is there something that can be done in such a case? Should the scripts behind marketplace apps be available somewhere (I just didn't see them)? Or are they usually closed off from the user?

According to https://azure.microsoft.com/de-de/blog/container-apps-now-available-in-the-azure-marketplace/ it seems to be possible to define the storage-size when a container-app is chosen.


Is there any way of getting code coverage with molecule? I'm currently working on testing Ansible roles with Molecule. I have one role with 2 scenarios that are executed in Jenkins but just before that, those are submitted to a SonarQube analysis. So I installed the YAML plugin for it to recognize the files, but now, I am asked to add code coverage to these reports.

I'm wondering if there's a way of having code coverage using Molecule with its default stack (Docker/Testinfra)?

You can use another verifier. Inspec should have a reporter that will report in the correct format. https://molecule.readthedocs.io/en/stable/configuration.html#inspec


Jenkins: Re-use repository with reference option and do changes on it I would like to setup a Jenkins cluster where the slaves are building an Android ROM with some parameters like branch (e.g. lineage-16.0) and some optional patches. The whole repository will consume around 100GB.
Let's say i want to build branch lineage-16.0 with patch-A and patch-B and in parallel build branch lineage-16.0 with patch-A only and in parallel build branch lineage-17.1 with no patches and so on.
I just read about the --reference option in git (which can also be used in repo).
Is it possible to use it this way or will changing the branch and applying patches modify the reference clone as well as it only uses pointers?
I need to save disk space and network traffic somehow.
If this is not possible is there any other way without duplicating the 100GB for each jenkins job?
I'm also open for other free tools than Jenkins.

If this is not possible is there any other way without duplicating the 100GB for each jenkins job?

To avoid frequent clones of the repository, you can put a single clone of the repository on-disk on your executor nodes outside of the normal workspace path.  Then, in your Jenkins job, you can cd to that directory with the dir step, update the repo, and checkout the branch you want, e.g.:
dir('/path/to/repo') {
  sh('git fetch --all')
  sh('git checkout origin/lineage-16.0')
  // your build steps here
}

However, this has the major drawback that the one repo will be shared across all of your jobs, which means you cannot run those jobs in parallel without risking corruption.  It sounds like running jobs in parallel was one of your requirements, so this may be a no-go.
You could probably get parallel builds to work by using a copy-on-write filesystem so that the repo appears to be in multiple independent directories but only actually has to be cloned once, but I've never tried something like this, so I can't say how well it would work.

Why is your "source repo" 100GB?
Admittedly, the article is several years old, but MS reported the windows code base repo was the largest known repo at the time and was 300GB.
Is your source really that big?  Your Git repo should NOT contain derived artifacts (.class, .jar/war, .o, .dll, .exe, etc), nor 3rd-party libraries. There are other tools for that.   You can use a .gitignore file to exclude these file types from being checked into your repo.
If you check in a binary file, Git will generally store a complete copy (generally non-compressible) of every copy you check in. I have seen developers check-in not just 3rd-party OSS jars in multiple locations in the same repo and same jars across multiple repos, resulting in massive duplication and wasted space, but then keep checking new versions. I've seen both parts of the Weblogic installer ( 2x 600 MB+) checked-in. And found one which was continuously checking in the Docker image he built which included 3party installers, making his repo almost 1GB larger every check-in!
You don't mention language, but the standard tools for Java libraries has been Nexus and Artifactory. They now handle as extensive range of language artifact types as well.
These tool also create local caches of the artifacts they retrieve so you no longer have to go to the remote host across the network if they have not changed.
Git-LFS is an excellent extension to handle large, non-derived binary files (audio samples, videos, datasets, and graphics).
Look at using BFG to remove unwanted artifacts from history and compress the repo.
As documented in the "largest repo ever" article, MS have contributed the "VFS for Git" to deal with really large repos if that really is the only solution available to you.
This is not an extensive how-to, just advice, "I think you're doing it wrong and that's why it's a PITA", plus some pointers to tools. You need to review your entire approach. That may include other tools like Ansible or Puppet to manage your infra configuration as well.
Summary: Manage your infrastructure as code and keep only source code in your source code repositories. Or just get a faster network.


Access vault secret from bash script I am looking for help on a direction on where to go from here. I have a Hasicorp Vault server running and accessible over DNS (local). I need to have a script get a secret from it then use that secret. I am not sure where to go from there. I know I want to use a bash script for this but past that I don't know what to do.

Until now I have been using curl to test it.

Suggestions?

The vault docs mention a -field parameter for the read subcommand.  So you should be able to put this into a shell script:
SECRET=$(vault read -field foo secret/mysecret)

Other vault docs use the vault kv get in the same way so you might try:
SECRET=$(vault kv get -field foo secret/mysecret)

Vault has a REST API available for reading.

https://www.vaultproject.io/api/index.html

In your script, just curl the API.


How to install Certbot via Ansible playbook? This is how I install Certbot in a Bash script:

apt-get update -y && add-apt-repository ppa:certbot/certbot -y && apt-get update -y
apt-get upgrade nginx python-certbot-nginx -y

That's the playbook syntax I found to install Nginx in default conf:

- hosts: localhost
    connection: local
    become: true

tasks:
    - name: install nginx
    apt: pkg=nginx state=installed update_cache=true

Is there no Ansible task to install Certbot as well, instead the above "heavier" Bash code?

Here is the direct Ansible translation of your bash install script:

- apt_repository:
    repo: 'ppa:certbot/certbot'

- apt:
    name: "{{ item }}"
    update_cache: yes
  with_items:
    - nginx
    - python-certbox-nginx

chipping in with ansible task to install letsencrypt certbot on debian stretch

---
- name: Add "stretch-backports" to APT sources (required for Certbot)
  become: yes
  apt_repository:
    repo: deb http://httpredir.debian.org/debian stretch-backports main
    state: present
    update_cache: yes

- name: "Check LetsEncrypt/Certbot is installed"
  become: yes
  apt:
    name:
      - certbot
      - python-certbot-apache # https://github.com/certbot/certbot/issues/3854
    state: present
    default_release: stretch-backports
    cache_valid_time: 3600
    update_cache: yes


Unable to get value of password parameter into another parameter In Jenkins pipeline parameter say Branches of type Active Choice Reactive Parameter, I've to auto-populate branch names hence I've following command, which I can execute in parameter's Groovy script -
svn ls <url> --non-interactive --no-auth-cache --trust-server-cert --username <username>--password <password>

username can be retrieved using User.current().getId()
I've Password parameter of Password Parameter type. To use value of this parameter in Groovy script of Branches, I've added Password parameter in Referenced parameters of Branches
But I cannot see its value reflected in Branches parameter. Can you please suggest how this can be handled?
Or any other way to pass Password value or authentication details so I can execute svn command.

Could credentials binding plugin be the answer?
The idea is to add svn user and password to Jenkins credentials store and then use it in the pipeline.
withCredentials([
    usernamePassword(
        credentialsId: 'mySvnLogin',
        usernameVariable: 'USER',
        passwordVariable: 'PASSWORD')]) {
    sh '''
      svn ls <url> --non-interactive --no-auth-cache --trust-server-cert --username $USER
 --password $PASSWORD
    ''''


Push Jenkins Build status to Github and Trigger Jenkins build from Github Webhook FYI - New to DevOps. Please bear with me. Thanks

I have a webhook created by Admin on Github Repo to trigger Jenkins build.
I created a token from my Github(I am not admin on the repo) and used it in Github Connection in Jenkins Configure.

Now if I commit when the token credentials are selected in Github in Jenkins Config, the Jenkins build is not triggered automatically. When I click on Build now the job is triggered and job status is pushed to Github.

If I don't select any credentials in Github in Jenkins Config, the jenkins build is triggered on commit but the build status is not reflecting in Github.

There is a manage hooks options also in Github in Jenkins Configure but that probably would require a token from Admin on repo?

Can you guys guide me on what should I do now to
1. Trigger the Jenkins Build on commit.
2. Push the build status to Github.

Thanks

Triggering Jenkins using GitHub webhook should be a straight forward process. This article would help you.

Found the issue. Certificates were needed to be installed which was blocking connection to Github Enterprise. Once I installed those, the setup is working fine now.

Thanks Everyone. :)


Is Docker right for my use case? My company has a system we sell that consists of basically a mini-computer "Smartbox" that is running Ubuntu 12.04. This box runs a Django application plus a number of different upstart processes related to it. Not much else. We have thousands of these boxes out on the field. We manage the package dependencies, process registration, etc. through a deb package with varying degrees of success.

We need a way to efficiently and robustly push updates out to our users out in the field. We also need something that as we upgrade the OS (we are way overdue for a Ubuntu upgrade as you can tell) we can feel relatively secure about our packages "just working".

I don't know much about Docker, but when I first heard about our problem (I'm a new hire), Docker was my first thought. But the more I thought about it I felt like maybe it wasn't, as these boxes are ours we control the OS on it which is a big part of the value proposition of Docker, or so I understand. So if we KNOW our boxes will always be Ubuntu and we basically just have a Django app plus some processes to run, is Docker any better than a deb package?

TL;DR: Docker vs deb packages for a distributed appliance that will always run Ubuntu so platform independence is not that important.

I'm not 100% sure I understand from the question, but it sounds like the Docker solution would be to go from having an (physical?) appliance with an OS and your app installed on it, to having an appliance with an OS and Docker on it, running a single container with your app in it. That doesn't obviate the need to update the OS in the host, and it adds a layer of complexity (and more updates to contend with, as you'll now have to keep Docker and the OS patched) with no readily-apparent benefit as far as the specific areas mentioned in the question are concerned.

However, if you're talking about going from a virtual appliance to a Docker container, that could potentially smooth things out for you, but it also adds Docker as a dependency for your product; you're shutting out anyone who isn't using Docker and doesn't want to add it to their stack just to use your product. You could continue to support those that don't/won't use Docker by continuing to ship the (now "legacy") virtual appliance as before, but now you've just doubled your workload because you have two distributions to support instead of one.

Docker sounds reasonable to me, as you could make and test changes to the container in house, and then depending on your release process, restart the containers always pulling :latest or something similar which would provide a tested upgrade.

Considerations you would need to deal with include data storage as containers don't retain changes on restart, so you would want a data volume. There are probably a lot more considerations you would have as well once you dig into it. The system that I am currently working with (all docker-based) has been in development for over a year now and we are still finding areas in which we need to make changes to the container, configurations, etc.


What is the term for a deployment designed to make a future cutover seamless? I'm currently working on a new major version (v2) of an application meant to replace the current production version (v1). This redesign carries with it changes to the front-end, public API layer, internal API layer, and database — the whole kit and kaboodle.

While it is easy to have active v1 and v2 front-ends and public API (which I'll call the "front half" for brevity's sake) simultaneously in our lower environments, for various reason, this is NOT possible for us to do with the internal API and database (which I'll call the "back half").

The v2 redesign necessitates changes to the back half that cause breakage in the v1 front half. However, in this particular instance, by making a minor change in the v1 front half, this breakage can be avoided. As such, it is possible for us to deploy this v1 "shim" along with the changes to the back half to production. When the time comes, we will be able to cut over to the v2 front half without the back half needing any further deployments.

Is there a name for this particular type of deployment, where a shim is deployed to an existing app to make it work with an updated back end, with the goal of making a future cutover seamless?

Taking my definition of blue/green deployment from this informative blog post, I believe this is a distinctly separate kind of deployment.

... by making a minor change in the v1 front half, this breakage can be avoided.

The above excerpt of this queston reminds me of something similar from around the late 90s, say 1996-1999 (when zillions of ITers were preparing for Y2K ...). In those days, new software releases of huge software vendors would come with release notes that included things like either of these:

"this release is to make this software Y2K tolerant " (= allows the software to contnue to work as of Jan 1, 2000, at least for a while, though it would keep showing dates with 2 digits for the year, like 00, and not yet 4 digits, like 2000).
"this release is to make this software Y2K ready " (= allows the software to continue to work as of Jan 1, 2000, without any (known) limitation because of the change of century from 19 to 20).

Y2K tolerant softwares would typically also include some sort of magical cut-off value, e.g.: years from 70 (when even MS DOS did not exist ...) to 99 were assumed to be from 1970 to 1999, while anything from 00 to 69 were assumed to be from 2000 to 2069. Such assumption was also why such softwares were required to be superseeded by some Y2K compliant release later on (in which those assumptions were obsoleted).

With the above in mind, the term you are asking for could be something like a "V2 tolerant release", which at some point in the future would then be followed by a "V2 ready release" (= your actual V2 release).

"Compatibility layer" sounds good to me.


DevOps SOX Compliance I'm coming from the dark side, compliance, and am looking to gain better knowledge on DevOps & how to implement SOX controls on these processes. I'm hoping to build a good knowledge base in order to be able to recommend controls that will provide good assurance to the auditors while minimizing the impact on the actual operations. Also, so that when I meet with our internal DevOps teams, I can speak knowledgably on this.

I've done some research through ISACA whitepapers and various other sources:

https://www.oreilly.com/learning/compliance-as-code
https://www.oreilly.com/webops-perf/free/devopssec.csp?intcmp=il-webops-free-product-na_new_site_compliance_as_code_text_cta
https://www.contino.io/insights/why-devsecops-is-an-auditors-best-friend
https://start.jcolemorrison.com/

Do you have any other suggestions on resources for building control frameworks for DevOps workflows? Or just DevOps in general?

what you are looking for is compliance and security for containers.  There are companies that can do this out the box for you, and even though I'm not advocating for either you can see what they offer and use that as a template.

https://www.twistlock.com/platform/container-compliance/
https://www.aquasec.com/use-cases/container-auditing-compliance/

As a HIPAA security officer and DevOps manager, I've been reviewing these processes.  The way to think of DevOps in your sense is probably the automation and culture of ship fast, fail early and fail often.  The goal is to not disrupt the pipeline from security and compliance.  Technically this looks like adding automated compliance checks in the pipeline before, during and after go live.  As automation gets built, you can replace the automation with a manual process'.  You can check out subjects like DevSecOps which is the same concept of ensuring compliance and probably closer to what you'll need to do.

here's an AWS diagram that might help https://aws.amazon.com/blogs/devops/implementing-devsecops-using-aws-codepipeline/

Have your read NIST 800 190? This document was co written by John morello who is the chief technology officer for twistlock, the company which does container security.

This NIST document helped me in understanding various attacks and what you can do. This might help you.