How to create the smallest working docker image every time? Aim: to create the smallest working docker images every time

Current

REPOSITORY          TAG       IMAGE ID            CREATED             SIZE
a-docker-image      latest    x                   42 minutes ago       1.92 GB

Attempt

Adding a cleanup step at the end of the Dockerfile:

#clean
RUN apt-get purge -y wget
RUN rm -r a-build-dir
RUN apt-get purge -y a-package

reduced the image size a little bit:

REPOSITORY          TAG       IMAGE ID            CREATED             SIZE
a-docker-image      latest    y                   2 minutes ago       1.86 GB

Discussion

I have built various docker images. Every time I try to decrease the size of the created image, but I always feel it is too large. I am looking for a script that has already been created by someone on github that removes all superfluous packages from the image so the size of the created image will be as small as possible.

As I said I always try to reduce the size of the image, but I want to apply this consistent so that every image that I create from now on will be as small as possible.

Question

How to create the smallest working docker image every time?

A Dockerfile creates a new layer for each of the commands in the file. Since layers are well, layered on top of each other - you cannot remove files that a previous layer added. This is why when you install packages, or download files, or create builds each in a separate command - these are still there in the image, even if in a future layer you removed them.

So if you just change this:

RUN apt-get update -y
RUN apt-get install -y wget a-package
# ...
RUN apt-get purge -y wget
RUN rm -r a-build-dir
RUN apt-get purge -y a-package

To this:

RUN apt-get update -y \
    && apt-get install -y wget a-package \
    && mkdir a-build-dir \
    && wget http://some-site/very-big-source-code.tar.gz \
    && tar xzvf very-big-source-code.tar.gz \
    && do-some-compilation \
    && apt-get purge -y wget \
    && cd .. \
    && rm -rf a-build-dir \
    && apt-get purge -y a-package

You will get a much smaller image.

Another option, is to squash the image after you built it.
Q: How does the new docker --squash work?

Yet another option, is to choose a slim base image. For example, images that use Alpine Linux as their base instead of Debian, take just 10-15mb instead of 180-250mb. And this is before adding your own application and data. Many official base images on Docker Hub have an alpine version.

Use docker ps to check the current running containers. For example:
FROM ubuntu16

MAINTAINER sreeni (email/domain)

RUN apt-get update

RUN apt-get install -y nginx

ENTRYPOINT [“/usr/sbin/nginx”,”-g”,”daemon off;”]

EXPOSE 80 (port)

Use below docker command to run the container:
docker run -d -p 80:80 --name web server ubuntu16

After that, check localhost or ip address:80 (open browser and check)


Why it is recommended to run only one process in a container? In many blog posts, and general opinion, there is a saying that goes "one process per container".

Why does this rule exist?
Why not run ntp, nginx, uwsgi and more processes in a single container that needs to have all processes to work?

blog posts mentioning this rule:

"Single-process-per-container is a recommended design pattern for Docker applications."
"Docker is only for creating single-process or single-service containers."
"better to use one process per container"
"Run a single service as a container"
"One process per container"
"one process per container"

Lets forget the high-level architectural and philosophical arguments for a moment. While there may be some edge cases where multiple functions in a single container may make sense, there are very practical reasons why you may want to consider following "one function per container" as a rule of thumb:

Scaling containers horizontally is much easier if the container is isolated to a single function. Need another apache container? Spin one up somewhere else. However if my apache container also has my DB, cron and other pieces shoehorned in, this complicates things.
Having a single function per container allows the container to be easily re-used for other projects or purposes.
It also makes it more portable and predictable for devs to pull down a component from production to troubleshoot locally rather than an entire application environment.
Patching/upgrades (both the OS and the application) can be done in a more isolated and controlled manner. Juggling multiple bits-and-bobs in your container not only makes for larger images, but also ties these components together. Why have to shut down application X and Y just to upgrade Z?

Above also holds true for code deployments and rollbacks.

Splitting functions out to multiple containers allows more flexibility from a security and isolation perspective. You may want (or require) services to be isolated on the network level -- either physically or within overlay networks -- to maintain a strong security posture or comply with things like PCI.
Other more minor factors such as dealing with stdout/stderr and sending logs to the container log, keeping containers as ephemeral as possible etc.

Note that I'm saying function, not process. That language is outdated. The official docker documentation has moved away from saying "one process" to instead recommending "one concern" per container.

The process I'll called as service here, 1 container ~ 1 service, if any of my service is failed then I'll only spin up that respective container and with-in seconds everything is up again. So, there won't be any dependencies between services. It is best practice to keep your container size less than 200 MB and max 500 MB (exception to windows native containers are more than 2 GB's) otherwise, its going to be the similar as virtual machine, not exactly but, performance suffice. Also, take into consideration few parameters as scaling, how could I make my services resilience, auto-deploy, etc.

And, its purely your call how you need to make your architectural patterns like micro-service in polygot environment using the containers technology that best suite your environment and will automate the things for you.


How to transition from a complex branching reality to a single-branch model? In large organisations, using the waterfall methodology typically results in very complex branching structures (aka branch spagetti).

What branching strategies can be used to transition from a complex branching reality to a single-branch model like trunk-based development?

Update:

To clarify, the question is about the migration/transition itself, not about the before and the after methodologies, which are pretty clear.

It can't really be "at EOB today we're still waterfall with gazillion of branches but tomorrow 1st thing we'll switchover to trunk-based, single-branch CI".

Because you mention waterfall, I understand that the numerous branches you are alluding to are feature-branches rather than maintenance-branches.

In this setup, I also assume that these branches are created according to a waterfall plan that tries to minimise conflicts.  This implies that the goal of the development is to produce several distinct products.  When using a single-branch development model, it is important to also work on a single product. If several products are simultaneously developed in a single-branch development model, it effectively “glues” together versions of theses products, so that we might have in version a of the repository a healthy product X and a buggy product Y, while in version b the product X has a regression and Y a bug fix, but we have no version where both X and Y are healthy.  Such a situation would force us to consider X and Y as being developed in distinct repositories, which is a hint that they should be.

Therefore, the first steps should perform a repository split:

Arrange the repository so that it is easy to split it into several small repositories. For instance, rearrange the current repository so that each top-level directory corresponds to a repository you want to create in the future. Doing so, you can continue to use the branch-spaghetti discipline everybody is familiar with.
When step 1 is completed, refine the branch-spaghetti discipline by requiring that any single branch can only touch files in one top-level directory.
When each branch complies with step 2, perform the split. Developers can easily convert their pending changes to patch a single repository, just by removing the first-level of the path.

Now that the split has been performed, you can start working on the branch discipline itself.

Introduce programming techniques helping the development of short-lived branches. Branches being short-lived is a crucial aspect of all single-branch methodologies. One of their goals is to reduce the time spent on merging and debugging long-lived branches. A popular technique is the introduction of “feature-flags” where a “factory” uses a configuration flag to either produce the historic version of an object or the new, initially partially developed, version of that object.
By now, you have zillions of repositories with only a few branches in each, and you can turn the “we globally adopt the trunk-based development discipline” button, without seeing the original branch-spaghetti mountain collapsing on the trunk.

The actual split of repositories might be optional, but you would then have to adopt policies that cleanly delineate the allowed scope of each submitted patch (to limit the risk of conflicts when merging changes in the main branch). Reducing the overhead bound to conflicts is one of the goals of single-branch model methodologies, so I assume this is relevant in your context.

For large organisations with typical 4 weeks sprint cycle Git-Flow is preferred approach because
You get benefit of Feature branch
Master production ready branch is always deployable
Also, master branch is kept clean from unwanted commits by following two commit cycle (from feature to Develop and Develop branch to Master).

Moreover, branching is also determined by frequency of production releases. For frequent deployment to production its better to have a Feature branch or Centralized model. In this case overhead of managing branches is shifted to robust testing in lower environments to maintain production stability.


How to check available volume provisioners in Kubernetes cluster? Is there a command or script that one can use to see or list enabled/available volume provisioners in a Kubernetes cluster?
We have an on-premise cluster and I came to know the hard way that 'local' provisioner has already been enabled in the cluster. So I am trying to see if there is something we can do upfront and set up a few of the provisioners for use by the containers/applications.

Came to know that there is no way to know this. Somewhere in the documentation I read that system will error out if the referenced provisioner is not present.


What's the difference between DevOps and Automation? I see that whenever someone does DevOps, it's mostly about automating things like deployment etc.

But where does automation end and DevOps begin?

A big part of DevOps is making it possible to release very often. That comes with automated build, automated testing, etc. You can say that to achieve its goals, DevOps need to use automation to be efficient.

Here's how DevOps and automation are related. DevOps is not just automation, there's more to it. Conversely, automation is not exclusively used by "DevOps people". A lot of automation was taking place in IT before DevOps came around.

Please don't consider to above diagram to represent all that is DevOps, or all this is automation. It is to help the reader picture how the two concepts are related.

Automation and DevOps are unrelated.  DevOps is more like combined engineering where the Developers of a site or service are all the Operators of that site or service.  Why is this novel?  In my experience the first thing the Ops team did when anything more exciting than a network blip happened was to call the Dev team.  Why did they do that?  Because all the Ops team did was monitor and keep a list of dev phone numbers to call.

Notice I've said nothing about automation.

Automation is about repeating success.  If I do steps a,b and c and process X always works, then steps a, b and c are good candidates for automation.  Then I can use the time for what used to be a manual process to do things that make me more money.  Automation is successful when it's simple.  Deploying new releases, running integration tests, torquing a nut on a bolt, backing up data, balancing credits vs. debits, etc. are all great candidates for automation because the steps are repeated whether by a person or a robot.

Note: What is new is that Developers are also the operators. There is not another group. Cooperation in my case was rare. If there was nothing in the Troubleshooting Guide (aka TSG) then you were guaranteed a phone call. In my experience Ops was the first call in case of backhoe. Problems between services were out of their wheelhouse.


Ansible Playbook Localhost Invocation Working Directory I want to run an ansible playbook locally (host is localhost, like -i "localhost,") and inside the Playbook obtain the working directory from which the ansible-playbook command was invoked.

However, when I read out the current working directory in a playbook task, I get the directory of the playbook itself.

Is there any variable I can refer to to obtain the working directory on the controlling host (which is localhost) in my case?

I can inject via a variable -e but I would prefer to have a fall-back that if that variable is not set I can obtain it from the ([local]host) environment.

You can use env lookup plugins. Plugins are always evaluated in the context of "parent" ansible process on the control host.

- debug:
    msg: "{{ lookup('env', 'PWD') }}"

More lookup plugins can be found here: http://docs.ansible.com/ansible/latest/user_guide/playbooks_lookups.html

I'd use the shell module to run a simple pwd then use register to capture that

name: current_working_directory
    shell: pwd
    register: current_working_directory

you can then reuse this anywhere

{{  current_working_directory.stdout }}


How to solve `no live upstreams while connecting to upstream` when trying to rproxy jenkins using jwilder/nginx? https://github.com/jwilder/nginx-proxy

nginx.1    | 2018/03/04 22:24:54 [error] 102#102: *276 no live upstreams
while connecting to upstream, client: 83.85.143.226, server:
jenkins.releasesoftwaremoreoften.com, request: "GET / HTTP/2.0", upstream:
"http://jenkins.releasesoftwaremoreoften.com/", host:
"jenkins.releasesoftwaremoreoften.com"

The docker-compose file looks as follows:

version: '3'
services:
  jenkins:
    image: jenkins/jenkins:lts
    environment:
      - VIRTUAL_HOST=jenkins.releasesoftwaremoreoften.com
      - VIRTUAL_PORT=8083
    ports:
      - 127.0.0.1:8083:8080
      - 127.0.0.1:50000:50000
    volumes:
      - /usr/local/jenkins/jenkins_home:/var/jenkins_home
    networks:
      - rproxy
      - jenkins

networks:
  jenkins:
  rproxy:
    external:
      name: rproxy

Another attempt to make it work, was changing the VIRTUAL_HOST to 80, but that did not work as well.

Navigating to http://jenkins.releasesoftwaremoreoften.com/ returns a 502.

After a spending a lot of time I decided not to continue with this approach and decided to use Jenkins helm.

VIRTUAL_HOST is meant to point to the internal port, in your case it should be 8080 instead of 8083


CodeBuild - multiple input artifacts in AWS Pipeline I am building an AWS CodePipeline which depends on two source repositories. If the 'CodeBuild' step is specified with multiple Input artifacts, then you get the error: .. declares 2 input artifacts which is more than the maximum count. Documentation has a strict requirement for CodeBuild to have only a single input artifact.

Lambdas, on another hand, do not have this limitation. I've been looking for a way how a Lambda could "merge" two artifacts into a single one which I can pass to the CodeBuild then. Unfortunately, information on this subject is quite limited.

Does anyone have a working CodePipeline Lambda which would merge two Artifacts into one?

Note: Documentation about How to access artifacts.

We've had exactly the same problem - our build stage needs to use information from two separate locations.

To solve this in a generic manner, I've created two general purpose Lambdas - one that will merge artifacts, and one that will extract specific files from an artifact. In both cases a new artifact is created that can be passed on to a subsequent CodePipeline stage.

See https://github.com/tooltwist/codepipeline-artifact-munge

If it does not meet your exact requirement, it should be easy enough to tweak to do what you need.

You can review CodeBuild. The BuildScpec file can be put in the CodeBuild project.


Ansible - Copy multiple files I am trying to copy multiple files to the target like so:

  tasks:
  - name: "Copying files"
    copy:
      src: files/{{ item }}
      dest: /my/folder/
    with_items:
      - file.txt

The files directory is in the same directory as the Ansible playbook. However when running the script, I get this error:

docker: TASK [Gathering Facts] *********************************************************
    docker: fatal: [test]: FAILED! => {"msg": "failed to transfer file to /home/user/.ansible/tmp/ansible-local-123331or28t2a/tmp6st_fxv1 ~user/.ansible/tmp/ansible-tmp-1579990845.0734024-189911264147319/AnsiballZ_setup.py:\n\n\n"}
    docker:

Why is that? Why can Ansible not copy the files to the target machine?

Following you comment: no I cannot reproduce your problem. Since you did not provide the rest of your files (inventory, etc...) and I cannot guess where is your problem, here is a quick proof by example using the php:7.0-apache image.

The inventory (inventories/docker_test/hosts.yml)

---
all:
  hosts:
    docker_test:
      ansible_connection: docker

Launching the test container

docker run -d --rm --name docker_test php:7.0-apache

The test playbook.yml

---
- hosts: docker_test
  gather_facts: false

  tasks:
    # Your image does not have python which is mandatory for ansible
    # Please note this is for demo only. You should get python installed
    # by extending the image through a Dockerfile and build it.
    - name: Dirty low-level command to get python installed
      raw: apt-get update && apt-get install -y python3

    - name: Gather facts now we can
      setup:

    - name: Copy a dummy file
      copy:
        dest: /etc/apache2/sites-available/dummy.conf
        content: "# I'm a dummy file"

Playbook run and output

$ ansible-playbook -i inventories/docker_test/ playbook.yml

PLAY [docker_test] ********************************************************************************************************************************************************************************************************

TASK [Dirty low-level command to get python installed] ********************************************************************************************************************************************************************
changed: [docker_test]

TASK [Gather facts now we can] ********************************************************************************************************************************************************************************************
[WARNING]: Platform linux on host docker_test is using the discovered Python interpreter at /usr/bin/python3.5, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.

ok: [docker_test]

TASK [Copy a dummy file] **************************************************************************************************************************************************************************************************
changed: [docker_test]

PLAY RECAP ****************************************************************************************************************************************************************************************************************
docker_test                : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Use the following source code for copy multiple files on your client machine.

---
  - name: Copy data to the client machine
    hosts: hostname
    become_method: sudo
    become_user: root
    become: true
    tasks:
      # Copy twice as sometimes files get skipped (mostly only one file skipped from a folder if the folder does not exist)
      - name: Copy UFO-Server
        copy:
          src: "source files path"
          dest: "destination file path"
          owner: root
          group: root
          mode: 0644
          backup: yes
        ignore_errors: true

Note:

If you are passing multiple paths by using variable then

src:  "/root/{{ item }}"
If you are passing path by using a variable for different items then

src:  "/root/{{ item.source_path }}"


docker-compose.yml - make network name configurable via environment variable I have a docker compose file with env variables for stage specific configurations. As long as env variables are used in values in docker-compose.yml everything is fine, but the problem is with the network name, since it's specified in the tag.

networks:
  mynetwork.${STAGE_NAME}:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: ${STAGE_NETWORK_PREFIX}.0/24

Any chance to get the network name mynetwork.${STAGE_NAME} configurable from outside?

If you have envsubst installed (part or gnu gettext) you can use something like that:

export STAGE_NAME="xyz"
export STAGE_NETWORK_PREFIX="1.2.3.4"
docker-compose -f <(envsubst docker-compose.yml) ...

If your shell does not support process substitution, something like that would do the trick:

envsubst docker-compose.yml | docker-compose -f - ...

There is a switch -p in docker-compose which allow you to set this prefix to a custom value.


Installing azcopy in Gitlab Runner (Alpine Linux) How can I install azcopy and use it in my gitlab runner CI/CD script?

First I have two scripts that download the tar file and extract it.

wget -O azcopy.tar.gz https://aka.ms/downloadazcopy-v10-linux
tar -xf azcopy.tar.gz

However, if I try running executable ./azcopy_linux_amd64_10.1.0/azcopy I get an error saying /bin/sh: eval: line 102: ./azcopy_linux_amd64_10.1.0/azcopy: not found
I know this directory/file exists because I've ran ls commands.

I've also tried copying the binary to /usr/bin and/or /usr/local/bin but I still get the error: /bin/sh: eval: line 102: azcopy: not found

What am I doing wrong? How can I extract and run the executable, or otherwise install azcopy in my gitlab CI/CD script

This is a glibc dependency issue on Alpine.

This azcopy binary is compatible with Linux distributions that use glibc, GNU's C standard library. Alpine Linux uses a different libc implementation, musl-libc, which is generally not compatible to glibc. When trying to run non-Alpine-built binaries on Alpine, they'll usually fail to link since the glibc shared object, libc.so.6, is missing. azcopy: not found is actually the result of the dynamic linker failing to resolve the requires libraries.

You could verify by running ldd ./azcopy:

    /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)

Note the libc.so.6 dependency. On vanilla Alpine, there's only the musl libc binary, libc.musl-x86_64.so.1.

Fortunately, Alpine provides a light compatibility layer for glibc, through the libc6-compat package. It installs a libc.so.6 link which is redirected to the musl shared object, and adds some missing glibc interface functions. This seems to be enough to running azcopy.

Therefore, for running azcopy on Alpine, just install the libc6-compat package, using:

apk add libc6-compat

After doing the extract you should set it executable with chmod +x azcopy

You can then run it with a azcopy --help


Why could the encrypted Ansible files not be decrypted although the vault password file has been set? Follow-up to this Q&A.

According to this and this documentation it should be possible to define a default vault password file by specifying vault_password_file = ~/.vault_pass.txt in one of the following files:

* ANSIBLE_CONFIG (an environment variable)
* ansible.cfg (in the current directory)
* .ansible.cfg (in the home directory)
* /etc/ansible/ansible.cfg

Although the password file exists and the log indicates that the ansible.cfg is used that contains the definition, the decryption fails:

user@host$
Using /etc/ansible/ansible.cfg as config file
ERROR! Decryption failed on /path/to/ansible/group_vars/production/vault

Discussion

  Is the pass correct for this vault?

Yes. When ansible is run using --vault-password-file=~/.vault_pass.txt the decryption succeeds.

  Just so that we're clear - the name of the environment variable is ANSIBLE_VAULT_PASSWORD_FILE, correct?

No. When this variable is used it works, but the aim is to set it in the ansible.cfg by setting /etc/ansible/ansible.cfg.

  What Ansible version is used?

user@host:/dir$ ansible --version
ansible 2.2.2.0
    config file = /etc/ansible/ansible.cfg
    configured module search path = Default w/o overrides

  Does using a full path in the config works?

The decryption failes as well when the full path instead of ~, i.e. /home/user/.vault_pass.txt was specified.

I found an issue with Ansible/Python causing a failure to decrypt.

Specifically, Ansible 2.7.10 and 2.7.13 under Python 3.6.8 on Ubuntu 18.04 would not decrypt (using ansible-vault edit -vvvvv) a vault file that was identical to file that decrypted OK on macOS 10.13 using Python 3.7.2 with same Ansible versions.

Upgrading to Python 3.7.3 on Ubuntu (using apt-get install python3.7 on 18.04) fixed this.

Your issue may be different, but if on Python 3.6 or lower, I would try upgrading to 3.7.

According to the docs,

  You can also set ANSIBLE_VAULT_PASSWORD_FILEenvironment variable, e.g. ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass.txtand Ansible will automatically search for the password in that file.

Ensures that you are using the exact same case.


Monitor Jenkins node/slave health For a project I'm working on, we need to create a Jenkins node in an AWS environment. For this task,  we will work with the docker image, which is available on https://github.com/jenkinsci/docker-ssh-slave
. The Jenkins node will be responsible for executing AWS commands which can only be done from inside the AWS environment (not outside).
This Jenkins Master is running in our local environment. We will add the node as a permanent node in the Jenkins master. The Jenkins master is not responsible for setting up the Jenkins Node, since it has no permissions to do this ( this is managed in AWS ).
However, we would like to monitor the Jenkins Slave inside AWS to see if it it still running successfully. With other docker containers, we also expose a health check. If the health check fails, the docker container is treated as unhealthy, and is restarted accordingly.
Just like we do with the other docker containers, we would like to expose such a health port as well on the Jenkins Slave. If it is not responding anymore, AWS will restart the docker container, and we have a stable system.
We already explored the Nagios monitoring, but it looks like this is exposing the health in the master, and not directly in the slave.
Is it available, or how can we add such functionality?
Could you lead us in the right direction, or recommend plugins which need to be installed, ...

Can we open the default HTTP API of Jenkins in such a Jenkins slave (https://wiki.jenkins.io/display/JENKINS/Remote+access+API)
Another approach could be to add and start NGINX in docker and point the healthcheck to NGINX. But with this approach, we are not really checking the health of the the Jenkins slave.

To me it looks like you are trying to reinvent the wheel. As I understood you would like to restart a container if a health check fails. As you are using docker you should use a orchestration platform in my opinion like Fargate (ECS) or Kubernetes (k8s). I know from the latter that it is possible to define healthchecks, like readiness and health probes. If the health check fails, kubernetes will restart the POD (auto healing).

It could be possible that you would argue that k8s is not an option. If that is the case, you could also have a look to AWS OpsWorks. Although I did not try it myself yet, it is able to perform auto healing as well.

Take a look at this Jenkins Plugin: https://plugins.jenkins.io/slave-status/
It'll expose a healthcheck endpoint + other information like memory on port 3141 (you can configure a custom port). You can then ping that healthcheck endpoint and send the data to CloudWatch.


What is the difference between Sysadmin and DevOps Engineer? When applying for a job, usually you can find two types of similar jobs: Sysadmin Engineer and DevOps Engineer.

Both of them deal with server configuration and ensure the reliable operation of computer systems. It can be hard to tell the difference between the two. What are the main differences between them?

Mainly DevOps is not a role (when used as such it's more a buzzword than a real role).

DevOps is roughly an organization pattern aiming at breaking the silo between developers and sysadmins.
The main goal is to build teams with devs and sysadmins (along with testers usually) responsible for a product (application) from its definition, architecture decisions up to the maintenance in run of this product.
Each member of the team will be part of the decision on the whole life-cycle of the product, a dev will do some sysadmin tasks in production, and a sysadmin will participate  in the design phase of the product to avoid caveats from the infrastructure perspective for example.

At ideal, a sysadmin would also be part of the development team for the product, in real world sysadmin code more on the configuration around the product and monitoring solutions, but being able to voice concerns to other members of the team avoid a lot of misunderstanding on the deployment process.

There is going to be servers which you do not hear running in data centers. Everything is going to be software. Storage, network, systems, security, data centers; SDN, firewalls, NFV, storage, servers, etc. Sysadmins without a software development background, SDLC experience (I don't even mean scripting Perl, Powershell, etc) will probably vanish. Distributed, scalable and virtualized environments, which is mostly cloud, grow horizontal not vertically.

I dare to say Sysadmins grow vertical, DevOps (or OpsDev) grow horizontal. You can see the same pattern how microservices evolved from monoliths. I would rather choose DevOps engineer from software team not from operations/system team.

Because operations/system team just runs what software team builds.

Sysadmins do not build/compile Linux/FreeBSD/windows kernel etc. just as software engineers build/compile applications.
Sysadmins do not go through the software development life cycle (SDLC).
Sysadmins are not part of the production pipeline (CI/CD process).
Sysadmin start working after CI/Continuous Delivery/Deployment ends.

And if you break and assign Deployment/Delivery, it could be a broken pipeline
the software team is the creator system/operations team are the runners/caretakers mostly.

It sounds like there is no server/system to administer, no sysadmin need.

Serverless computing is a cloud-computing execution model in which the cloud provider acts as the server, dynamically managing the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity
Serverless computing

Someone from the software team already knows how to build, maintain even how to code (SRE vs DevOps/OpsDev).

I wonder why it is called DevOps but not OpsDev? is it related to the direction between the two?

*In the middle of nowhere, I did not start writing about software-defined storage, this is in reaction to a now deleted comment about it *

About software-defined storage

Software-defined storage (SDS) is a marketing term for computer data storage software for policy-based provisioning and management of data storage independent of the underlying hardware.
Software-defined storage
EMC announced its first ever open source product: Project CoprHD. CoprHD is a Software Defined Storage automation and management controller, and EMC’s recent decision to open source it is at the crux of our strategy for delivering greater value to global businesses as we enter into an area of growth and extreme change.
As the world’s leader in storage and information management, it behooves EMC to lead the way on Software Defined Storage (SDS).
CoprHD is an open source software-defined storage controller and API platform. It enables policy-based management and cloud automation of storage resources for the block, object and file storage providers
CoprHD


What is an artifactory? There are quite some questions and answers that mention "artifactory".

I wouldn't be surprised if it is somehow related to artifacts.

My questions:

What is actually an "artifactory" (in the context of DevOps)?
Why are artifactories used?

Artifactory is a product by JFrog that serves as a binary repository manager.  That said very often one will use a 'artifactory' as a synonym of the more general binary repository, much like many people use Frigidaire or fridge to denote the refrigerator regardless if it is a Frigidaire brand or not.

The binary repository is a natural extension to the source code repository, in that it will store the outcome of your build process, often denoted as artifacts.  Most of the times one would not use the binary repository directly but through a package manager that comes with the chosen technology.

In most cases these will store individual application components that can later be assembled into a full product - thus allowing a build to be broken in smaller chunks, making more efficient use of resources, reducing build times, better tracking of binary debug databases etc.

Here are some of the most popular package managers that can be managed using a binary repository:

Java: jar, ear, war etc has Maven and the official MavenCentral.  There are many other package managers that will use the maven binary repository format as well (ivy, gradle etc).
.Net: nuget for .NET components (DLL and EXE) but can also be used as a distribution mechanism under windows thorugh systems like Chocolatey.  Newer versions of Powershell can also leverage this to distribute powershell modules though the powershell gallery of which one could build a local distribution with a binary repository and a repository in nuget format.  Also check OneGet if Windows distribution management is of interest to you.
In JavaScript: we have npm which is one of the most popular, will require nodejs.
In python: there is pip and the official package index pypi, which one can also create a local instance through binary repository that will support the format.

This list is far from complete, just gives an idea of what's out there.

The binary repository can allow to host all of these under one roof, making their management much simpler for teams.  Note that you do not need a very large team to start reaping benefits from binary package management.  The initial investment is not very large and the benefits are felt immediately.  Especially now that more and more platforms, frameworks and languages are integrating this dependency management directly in them.
Their biggest advantage I have found however was to create an environment that your programmers will find natural and comfortable making it essential.  It helps you as a devops creating a solid tool-chain and it helps them making the overall experience fit naturally in their stack of choice.

As I said earlier there are many products out there that can serve as binary package managers, some more generic than others in their target usage, varying also widely in their accessibility and prices.

My personal opinion is that binary repositories are as vital a part of a well designed devops setup as the source code repository or continuous integration.

An artifact
Is something that is produced/generated/crafted out of a specific process

Jar out of a Java project build.
Question out of your mind
Car out of a factory
New song

A repository
Is a receptacle where things are persisted

Github for a Java project.
StackExchange for your daunting questions

IN THEORY, an Artifact-ory would be a repository of artifact where they are persisted and managed throughout their life cycle.
In the context of DevOps, Artifactory is a product that manages binary artifacts. It stores and manages different types (Jar, Python and npm packages, etc...) that you produce out of your builds and re-use at compile or deployment time.


How do I automate deployments with Kubernetes? Assuming I have Kubernetes deployed via Rancher and Jenkins is building new images and pushing them to DockerHub upon check-in of new code to GitHub, how do I automate getting new images deployed?

Another way to ask the question might be, “I used to use Octopus to manage my deployments. Is there something similar built in to Kubernetes or Rancher?” Ultimately, it’s this last gap that I’m struggling with.

As of you are using Rancher the easiest way would be to register custom Rancher Catalog and create an item for each stack/service you want to deploy.

Rancher Catalog is a Git repository with the following structure

Then in Jenkins you can create a job that would call Rancher REST API to deploy/update stack/service.

In its turn Rancher would pull latest version of the Docker image for this service and deploy it corresponding to Docker Compose file from the catalog.

Pros:

generic approach that might be used for almost every app
Jenkins itself might be deployed into Rancher environment, agents might be created in Kubernetes cluster

Cons:

development team should follow solid release strategy to be able to use generic builds
storage drivers are still under the question in alpha release of Rancher 2.0


What's the difference between "Ops Dev" and "DevOps"? I've heard both terms, but I can't differentiate between the two.

I've never seen ops dev used before this post.  Searching around I see things like this article that has it in the title but nowhere in the article.  Then there's this article from 2015:

  So in this way, when “OpsDev” happens, IT ops teams are helping developers do their jobs, but developers are also architecting applications with IT ops in mind, helping them utilize resources better. The days of developers simply creating applications and “throwing them over the fence” to testers and IT ops teams are over.

which seems to define OpsDev as "development in a company that practices DevOps" (or more cynically, "development done correctly").

Finally, there's https://dzone.com/articles/opsdev-is-coming :

  What Is OpsDev?
  OpsDev means that the dependencies of the various application components must be understood and modeled first before the development process begins. In addition, the consideration for infrastructure stability, environment modeling, security and audit/compliance measures are first and foremost. Application components are stubs and they do not need to be in their final forms. Secondly, the environments in which the components will be deployed for production must be modeled. Thirdly, the processes to deploy the various components to the target environments must be automated as much as possible.

Again, this sounds to me just like normal, modern development.

DevOps is a problematic term because it has many different definitions.  OpsDev, on the other hand, appears to have essentially no definitions.  From a personal perspective, I would mentally substitute "development" for every instance of "OpsDev" and ignore the term; it appears to be just another instance of people inventing new terminology to describe old concepts so that they sound more exciting.

In the operations team DevOps does not trust developers to make changes to the production infrastructure. Developers want to focus on their application and the value it delivers to the organization. Developers want to know the characteristics of the infrastructure but do not want to build it or operate it.

The operations team is absolutely critical to the success of DevOps methodologies.

The developers must be able to trust that the infrastructure has specific characteristics: characteristics like performance, connectivity, availability, and uniformity.

To enable this trust, I believe that the operations teams are going to need to become more like developers. I call this OpsDev.

OpsDev is not simply about having a scripted build for servers. OpsDev is about having a full infrastructure that is built and maintained automatically from a set of source-controlled configuration files, and then having development and test environments that are built using the same methodology.

Source: https://www.astroarch.com/tvp_strategy/devops-requires-opsdev-time-change-35139/


Would choosing the integration branch strategy be part of DevOps? In a mature DevOps organisation would choosing a product/product line integration branch strategy be part of DevOps?

What do I mean by integration branch strategy?

For example a product team has to integrate 20 features in a certain interval of time to meet some time to market release requirements.

One potential approach would be to use feature branches in a waterfall manner as they've deen doing for years - each feature is developed and tested in their own branch and only merged into the main branch (from which the release branch will be pulled) after metting the respective feature QA goals.

But the total integration time estimate - 20 feature branch merge windows multiplied by the average merge duration (from the team's branch merge historical data) is maybe too high.

Another approach is to pull say 4 intermediate integration branches, each of which being used by 5 of the features branches. This way all 20 features will be integrated into the 4 intermediate branches in a 5 branch merge windows interval. Which would be followed by a 4 other branch merge windows interval for integrating these 4 branches into the main branch, bringing the total integration estimate to 9 windows, much better than 20. Yes, some of the gain will probably be cancelled by higher degree of difficulty of last 4 merges.

Another approach would be to go straight for trunk-based development (true CI integration into the main branch) with practically no branch merges.

The process of selecting one such strategy is what I'm referring to as branch strategy.

Note: the actual SCM repository is irrelevant, it can be any combination of repos managed together as a single one, the term branch denotes just the common logical representation of whatever physical/actual repo branches exist in that combination.

Branching strategy is a very clearly a business decision in which the DevOps engineer should be the primary expert, explaining all the trade offs. There are some strong positions on this issue in the community, very clearly Jez Humble is on one side with his book of Continuous Delivery. You could see him talking about it here. Here would be a video by panel of experts on the topic. And here is my highly up-voted answer on StackOverflow specifically on topic of feature branches. So clearly this is a big topic for DevOps.

One thing that people often say (Jez Humble mostly) is that feature branches are evil. You will need a (n-1)(n-2)/2 merges [combination number n-1 over 2] for n independent feature branches. Which comes to some very high numbers at 10+ feature branches. And because merging is hard and no publicly available version control systems do it will, then the entire Software Industry is moving away from software releases and software on premises to Software as a Service, where the only way is forward and software is both developed and maintained for you by the vendor.

As far as I know, there is only one company that can do high quality, business critical software on premises, with maintaining 10+ years of past major releases, back-porting patches through multiple waves of code refactoring and capable of working regularly on dozens of major feature branches per release and thousands small features developed on their individual feature branches and being continually merged as full features and not individual commits. The code has also been released on 14 different architectures. That company is Oracle, the product is RDBMS and they have been practicing what is now called DevOps since at least 1990. Hundreds of code deployments a day back in 2001 would be pretty standard. The reason why they are able to do it is through rigorous coding standards, invariant test suite and a proprietary version control system I've spent 10+ years of my life developing. A system that can do 99% of all merges through automation.

Everyone else pretty much only patches the most recent major release version past a certain stage/size of a project or it takes extreme amounts of human resources to merge all the patches to all the possible release versions and extreme amounts of human or compute resources to test them all. For example in Linux kernel you would have a maintainer for X.odd development version and later stability branch maintainer for X.even version, but you would see not patches for X-1.Y versions. Sometimes Linux distributions would maintain kernel version in their LTS release, but only at high cost and with varying code quality among distributions. And that is a high visibility project with enormous resources poured into it.

The primary reasons for this discrepancy are differences in:

Adherence to Coding Standards and Code Styles
Complete coverage by invariant test suite, which can catch bugs for which no tests were written specifically.
Version Control System capable of high levels of Automation.

DevOps should take part in this decision, but so are developers, product managers, testers, support and possibly even sales and marketing to some degree.

Branch strategy might affect the final product release, so it shouldn't be a one team decision.


What are the advantages of dockerizing nginx and php in different containers? I just started working with Docker and Kubernetes and I've been watching a lot of stacks, in which some people build nginx+php in a single image and some build an image with nginx and another one with php (mounting the same path and enclosing both containers in the same deployment in Kubernetes).

What would be the advantages of building two docker images instead of installing both nginx+php in the same one?

PHP with nginx is usually done using php-fpm which is a separate processus.

Keeping the core idea of docker of one process (see end of answer for more details on this point) per container this makes sense to have the nginx process and php-fpm process in separate containers.

As the communication between nginx and php-fpm arise through fastcgi the php-fpm container can also be on a separated host and this allows using a cluster of php-fpm containers behind nginx.

After the wall of comment here's a little more background, docker documentation have paragraph about the idea that a container should have only one concern.

The main idea of a linux container (lxc) is to run a process in an isolated namespace at the cpu and memory level, docker add on top of this an isolation at the filesystem level.
The advantage is that the compromission of a process within this namespace won't allow to read memory of other processes and as such should prevent other compromission on the host.

While talking about nginx and php-fpm, they work in pair but each has it's own concern, nginx will do the HTTP part, routing, headers validation, etc. and php-fpm will do the code interpretation and return the html part to nginx. While it's usual to have both together serving a single application that's not mandatory.

Depending on context it may be easier to have a container including the whole stack for an application, on a developer workstation for exemple. But ideally for production use, try to keep the fewer interaction inside the container, having separated processes in the same container with supervisord brings its share of problem in term of zombie process and log handling (exemple story here for illustration purpose only).

So finally I'll quote the docker page with some emphasis:

  While “one process per container” is frequently a good rule of thumb, it is not a hard and fast rule. Use your best judgment to keep containers as clean and modular as possible.

There's no "silver bullet rule" which apply to everything, it's always a balance between the complexity within the container and the complexity orchestrating the containers themselves.

The advantage is : you can run multiple php-fpm containers in back-end, we call it PHP cluster, via number of ports.
Example port 9000, 9001, 9002 .. and so on


How do you simulate "sudo su " in Ansible? I have a server where my user is able to sudo to the root user without a password, and then the root user can sudo to a third user without a password.  However, my user cannot sudo directly to the third user without a password.

Ansible's become directive uses sudo in the traditional manner, i.e. the one that requires prompting for my password.

I've tried putting become_user: root on a block and become_user: <other-user> on a task inside that block, but it appears Ansible overrides the block's definition, rather than nesting the sudo calls as I hoped.

Note: this question has also been asked on StackOverflow, but since there's no solution I figured I would repost it here, which I think is the better site for it.

I met the same situation. I have login1 for my machine, passwordless sudo and login2, under which I should perform some actions. I did not solve it by ansible means. But I made such a workaround:

- name: "Install nvm"
  shell: sudo -u buildkite-agent bash -c "<my commands to be performed on behalf of buildkite-agent>"
  become: true

Such a case:

- name: "Install nvm"
  shell: sudo -u buildkite-agent <my commands to be performed on behalf of buildkite-agent>
  become: true

do not change home to buildkite-agent's home, i.e. uses /root as home.

For example you want to create a file you can use :

 name: Create file
 shell: "touch abc"
 become_user: <username>
 become: yes

This is equivalent to ::
sudo su username -c "touch abc"


How to disable ping thread on jenkins agent One of the jenkins slave is going down very often. Found RCA for this issue.
It's because of the Ping thread

In the logs I fetch from agent, I have:

Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1576130988405 hasn't completed by 1576131228405
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

I wanted to disable the ping thread on the agent. In the documentation it's mentioned as -Dhudson.remoting.Launcher.pingIntervalSec=-1

But I do not understand how to execute this on remote machine.

I followed this link. but that solution is for JNPL. I am launching the agent on the node via ssh.

In the web UI, from the home page, navigate to Manage Jenkins > Manage Nodes > Configure.  This page has all of the health checks that Jenkins performs to determine if an executor is up.  The check you are concerned about is called "Response Time".  You can disable the "Response Time" health check on this page.

From the help text on the same page:

  This monitors the round trip network response time from the master to the agent, and if it goes above a threshold repeatedly, it marks the agent offline.

  This is useful for detecting unresponsive agents, or other network problems that clog the communication channel. More specifically, the master sends a no-op command to the agent, and checks the time it takes to get back the result of this no-op command.

The executor thread is waiting on the channel to give it more data.

You got a little confused due to the fact that the ping thread has the name of the channel Object in its threads name (which is the same Object that the executor is wating for).


Aggregating JUnit test results from multiple projects We have a Jenkins setup which utilizes multiple projects for running various suites of tests for our application.

There is one 'parent' project, which is triggered by Git commits. This project in turn builds our 'child' project. Each child projects runs a series of defined tests.

Each project is setup with JUnit to output onto the build page the results of the tests.

In order to get a comprehensive view of commits across our entire project, you need to go to the build page of EACH of our 'child' projects to see the results.

What I would like to do is aggregate ALL the JUnit reports, from builds triggered by out 'parent' project across all the 'child' projects to one place so we can view all the test results together.

Does anyone have experience with something like this?

The JUNIT plugin has a built-in publisher for aggregating test results called "Aggregate Downstream Test Results". You need to make sure that the 'parent' and 'child' jobs are linked together with a downstream/upstream relation. This is usually done by archiving an artifact in the parent build and copying it to the children builds using the 'Archive Artifact' publisher and the 'Copy Artifact' buildstep respectively.

If you are able to use gradle, maybe use something like mentioned here:

Multi-module projects
or
this

Also you'll get coverage :)


Connect external Jenkins Master to GKE with Kubernetes plugin I find the documentation at https://github.com/jenkinsci/kubernetes-plugin quite unclear when it comes to step by step configuration of external Jenkins master.

What are the exact steps that need to be followed?
I have installed the Kubernetes plugin and I have tried to supply some cluster info, but I have got lost there.

For Name I have taken the output from kubectl config view --raw, specifically the context for the given cluster and the cluster name part.

For Kubernetes URL I have set the IP address from the Server field in the same output from above.

Kubernetes server certificate key - for this part I have initially tried to follow the same pattern from above and take the value of the certificate-authority-data field, but I got an "Error testing connection https://XX.XXX.XXX.XX: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input", so I have attempted to convert that value with echo "the field value here" | base64 -d > ca.crt and use that, but it failed again and I have also got it form the UI for GKE show cluster ceritficate section, all of them give me failure on testing connection with the error from above.

I couldn't really go past this point, but I see everything being quite unclear forward, so I would appreciate a step-by-step recommendation or any guide that I am missing.

You could install Jenkins using helm to install the k8s plugin as well.

https://github.com/helm/charts/tree/master/stable/jenkins


What methods are available for connecting many-to-many containers? I have two containers (Docker):

TensorFlow container
NodeJS container used as a frontend to the TensorFlow container.

Both containers have TCP listeners on a port specified using environment variables.

If I want to scale the NodeJS and TensorFlow containers, for example 2 instances of NodeJS and 4 instances of TensorFlow, what are my options for connecting them to evenly spread the load across the 4 instances of TensorFlow?

Currently I'm using the TensorFlow container name inside the NodeJS container, but I'm pretty sure that would only take advantage of a single TensorFlow container... Or??

The classic answer to balancing load is, of course, a load balancer. There are many different methods of doing this, but we're interested in ones that work well with dynamic containers.

This is one of the things an orchestration layer will address. In Kubernetes, for instance, you would create a Tensor Flow service, and scale it to however many pods you need (or use autoscaling!). Kubernetes then provides an ip for the service that kube-proxy load balances across the pods.

There are also service mesh options that replace this layer and perform a similar function.

You should assign a specific port to any containers.
For this configuration you can use docker-compose file such as following example:

version: '3'

services:
  TensorFlow-1:
      image: gcr.io/tensorflow/tensorflow:latest-gpu
      container_name: first-tensor
      ports:
        - "9001:8888"

  TensorFlow-2:
      image: gcr.io/tensorflow/tensorflow:latest-gpu
      container_name: second-tensor
      ports:
        - "9002:8888"

Then in the docker-compose.yml path, on command-line do it:

sudo docker-compose up

Or

sudo docker-compose up -d

NOTE: -d option is for run above command in the background.

Finally you have two tensorflow containers with 9001 and 9002 ports.


Jenkins: how to run task every 30 days Jenkins version is 2.235.5 (I should update it to next LTS version 2.249).
I configured Jenkins to run a job periodically every 30 days: Build Triggers → Build periodically → Schedule:
H 3 */30 * *

And now I see:

Would last have run at Saturday, October 31, 2020 3:20:39 AM EDT; would next run at Sunday, November 1, 2020 3:20:39 AM EST.

So it will be started again after 1 day, not 30 days…
How to run a job periodically every 30 days?
Update: history of Jenkins "predictions" for schedule above (job was created October 28):

Would last have run at Thursday, October 1, 2020 3:20:06 AM EDT; would next run at Saturday, October 31, 2020 3:20:06 AM EDT.

Would last have run at Saturday, October 31, 2020 3:20:39 AM EDT; would next run at Sunday, November 1, 2020 3:20:39 AM EST.

Would last have run at Sunday, November 1, 2020 3:20:43 AM EST; would next run at Tuesday, December 1, 2020 3:20:43 AM EST.

In short, it's not practically possible. This S/O answer is quite detailed.

Jenkins used a cron
expression, and
the different fields are:
MINUTES Minutes in one hour (0-59)
HOURS Hours in one day (0-23)
DAYMONTH Day in a month (1-31)
MONTH Month in a year (1-12)
DAYWEEK Day of the week (0-7) where 0 and 7 are sunday

cron syntax simply does not support every n days.
An impractical option wuld be to have a daily job with a Conditional BuildStep to calculate if this day is n × 30 days from a given fixed day and abort if not..
But every 30 days doesn't really align w/calendar anyways, so is that really the intent?

Jenkins- scheduled tab change syntax like...
. 30 * * * *
Example for understanding.... similar to cron job


What could be a valid definition of DevOps to introduce it to a novice? I've done / created lots of SCM related presentations, and now I'm trying to "upgrade" to a DevOps successor of it.

What I always try to do in my presentations, is to come up with an introduction slide which somehow includes the message I want to deliver (and which I then elaborate on in the rest of my presentation). When doing so, I try to answer my own question like "What would me 1 to 3 phrases be that I'd want to use if I got like 10 to 20 secs (only!) to explain it to somebody new to it?*".

I thought I knew what DevOps actually means, and what it is about. But I've seen some bizarre usages/contexts of DevOps (even on DevOps.SE ...). It makes me wonder if maybe what I think DevOps is, is completely wrong.

So what is generally agreed to be the definition of DevOps?

DevOps in a nutshell

From Wikipedia:

  DevOps (a clipped compound of "software DEVelopment" and "information technology OPerationS") is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes.

  It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.

From Overview:

  Venn diagram showing DevOps as the intersection of development
  (software engineering), operations and quality assurance
  (QA)

While there is no single "tool" for DevOps, but rather a set of tools, also known as a DevOps toolchain:

  Illustration showing stages in a DevOps toolchain

Illustrations of DevOps

Below are a few quotes from some of the questions on DevOps.SE, which all seem to somehow fit / confirm part of the DevOps description above:

From 'What is the difference between SRE and DevOps?':

  ... DevOps is about maintaining code ...

From (an answer to) 'How do I hire a good DevOps, fitting my company?':

  ... It is a common misbelief that “DevOps” is a role in a company. The word merely denotes an organisational shift in software companies ...

From 'Does my organization need adopt Agile Soft. Dev. before adopting DevOps?':

  ... there are still organisations which are not practicing Agile in their software development, who might be interested in adopting DevOps ...

DevOps is NOT a role

Below are a few quotes from some of the questions on DevOps.SE, which all seem to illustrate that DevOps is NOT a role:

From (an answer to) 'What is the difference between Sysadmin and DevOps Engineer?':

  ... DevOps is not a role (when used as such it's more a buzzword than a real role).

From (an answer to) 'How do I hire a good DevOps, fitting my company?':

  ... It is a common misbelief that “DevOps” is a role in a company. The word merely denotes an organisational shift in software companies ...

Devops is the development practice of writing applications whose business domain is operations. Where most application development focuses on building applications which do finance or health care or logistics or cat videos, devops focuses on applications which enable builds, deployments, monitoring, and metrics gathering.

The overriding goal should always be to enable decision-makers to become decision-takers. Imagine your bank's mobile application. When you request a transfer, it happens when you hit the button. You made a decision, then took the decision. Same thing with your operations. When the appropriate person decides some work is ready to deploy to production, they should be able to hit a button and the "Right Things Happen". Similarly, they should have all the necessary information they need to make correct business decisions.

It's not about giving business people shell access to servers - that's confusing purpose with implementation. It's about providing the correct knobs and levers with the right information and the right guard-rails to the right people so that decision-makers are decision-takers.


How to test a cluster of microservices? I took a look into Server Spec and it says it clearly:

  Remark: serverspec test suites are meant to be run against a single machine  (or docker container). In other words, you should not try to issue a single rspec command that would harvest and run tests against multiple machines or containers. You need to issue one rspec command for each of them.

So I can do test per-server, which is good for some cases. But the problem is the following: my microservice architecture has a auto-discovery service, so some services are known after querying it. Is there any project to express this? I know I can tool around with ruby (or python if I choose  testinfra or other)

Something integrates with Consul with serverspec or similar will be great, in 2014 people were looking for this, does anyone know of any project that tackled this issue?

If nothing exists to easily mock Consul, have you considered spinning up a Consul service along with your other microservices? Consul can run in Docker, so this should be fairly straightforward if you use docker-compose and your apps can be configured to point to a different Consul endpoint.

Once your docker-compose is configured to spin up all the infrastructure/microservices required for your test, you will be able to use whatever tools you're familiar with to test your microservices, e.g. Selenium.

locally you can test against kind (kubernetes in docker) cluster or k3d which is tiny k3s cluster in docker.
Just spin up the cluster, install bundle of microservices there and run the tests.
Have a look on https://github.com/AbsaOSS/k3d-action which demonstrates spinning such cluster.


Kubernetes configuration with yaml anchors I have a deployment.yaml file and want to reuse the environment for all my deployments like this:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: beat
spec:
  selector:
    matchLabels:
      app: beat
  template:
    metadata:
      labels:
        app: beat
    spec:
      containers:
      - name: beat
        image: myimage
        command: ["celery", "-A", "wsgi.celery", "beat"]
        env: &default
        - name: FIRST_ENV
          value: my-value
        - name: SECOND_ENV
          value: another-value
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flower
spec:
  selector:
    matchLabels:
      app: flower
  template:
    metadata:
      labels:
        app: flower
    spec:
      containers:
      - name: flower
        image: myimage
        command: ["celery", "flower", "-A", "wsgi.celery"]
        env: *defaultenv

But it seems like kubectl apply -f deployment.yaml won't work with YAML anchors.
error: error parsing deployment.yaml: error converting YAML to JSON: yaml: unknown anchor 'defaultenv' referenced

Is it possible to use YAML anchors or is there another preferred approach of how to reuse repeating blocks for k8s configuration?

YAML anchors are supported, but only for the same YAML file. You can't create the anchor (&) on a deployment file and reference (*) the value on another one.
If you want to share ENVs values across multiple Deployments, you can create a ConfigMap with the ENVs and use the envFrom spec. Example:
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-configmap
data:
  FIRST_ENV: my-value
  SECOND_ENV: another-value
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: beat
spec:
  selector:
    matchLabels:
      app: beat
  template:
    metadata:
      labels:
        app: beat
    spec:
      containers:
      - name: beat
        image: myimage
        command: ["celery", "-A", "wsgi.celery", "beat"]
        envFrom:
        - configMapRef:
            name: my-configmap
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flower
spec:
  selector:
    matchLabels:
      app: flower
  template:
    metadata:
      labels:
        app: flower
    spec:
      containers:
      - name: flower
        image: myimage
        command: ["celery", "flower", "-A", "wsgi.celery"]
        envFrom:
        - configMapRef:
            name: my-configmap

The preferred way is to use helm and create a chart. Helm practically generates your manifests based on templates.


Is there an image on dockerhub that is able to run make? When one searches for make on dockerhub multiple images are returned, but none of them seems to be able to run make.

GNU make can be found in the gcc container.

  Compile your app inside the Docker container
  There may be occasions where it is not appropriate to run your app inside a container. To compile, but not run your app inside the Docker instance, you can write something like:

  $ docker run --rm -v "$PWD":/usr/src/myapp -w /usr/src/myapp gcc:4.9 gcc -o myapp myapp.c

  This will add your current directory, as a volume, to the container, set the working directory to the volume, and run the command gcc -o myapp myapp.c. This tells gcc to compile the code in myapp.c and output the executable to myapp. Alternatively, if you have a Makefile, you can instead run the make command inside your container

In my opinion there shouldn't be an image to run make, because if you want to build something from source you should install the dependencies and configure before run make, so I think it is just a command and you don't need an image to install it, even make is not a service to be in separate image, it must be used in Dockerfile to help you building your service from source during the image build time and not create a specific image for it.


How to find the constraint in a software development shop? Let us define a constraint as anything that prevents the system from achieving its goal and then look at the goal of DevOps in an organization.

A good definition is "DevOps enables a fast flow of features from development to IT operations to the customers." Improving DevOps can be achieved by removing impediments to flow.

First, we must find these constraints in the flow. And maybe there are multiple obstacles, how can we decide on which one to remove first?

A famous quote from Dr. Eliyahu M. Goldratt is

  Any improvement not made at the constraint is an illusion

This constraint, once found, is an important focusing mechanism for management of the system. So, how can you identify the constraint in an IT organization? And what to change about it to get the benefit of better flow?

To identify a constraint in any process is a relatively simple task. Work moves from a person to person throughout the organization and it will simply pile up in front of a constraint. You can look for people or teams with the highest number of blocking unresolved issues for example. It depends on how you track the flow of work, but if you do track it, then identifying constraint is usually not too difficult. It helps to know the value chains for your organization and simply follow along them.

Constraints can be of three types:

People: A particular skill or knowledge unique to an individual.
Policy: A written or unwritten policy or process, which limits the work either by increasing latency or reducing throughput.
Equipment: Particular computer system can limit the speed at which you can pass work through it or the number of people that can use it at the same time.

In regards to software, the Software Delivery Process is often a constraint and thus Continuous Integration / Continuous Delivery is such a powerful way to elevate this constraint and subordinating the organization to this constraint is often needed to increase the throughput of work. By automation in Software Delivery you remove slow people and often costly policy obstacles and approvals. Equipment in form of Computer System upgrades is a relatively small cost comparatively to the other two.

QA is another example of possible constraint in the process. Manual quality controls are work intensive and even automated testing can be constraint by equipment, in case the software is to run on expensive hardware. But the most common constraint in QA is policy, which tends to snowball with ever problem. The latency of the QA process and the small throughput can lead to larger units of work being put through, which can push back the cost into code integration.

Unfortunately there is no one answer here. If you ask each member of your team, they will probably all have different answers. Instead of trying to a root cause analysis on something as generalized as "culture", try looking at return-on-investment instead. A fix to something near a bottleneck will usually have a better RoI than one further from the bottleneck, but if the further one will take 5 minutes and get you 90% of the benefit then who cares, do the needful.


What are tools that lend themselves for backup and recovery in a CI/CD environment? I have used tar, dump, Amanda, Legato (EMC) Networker and other backup software for making backups of systems.  However, I don't know the best tools for making backups in a CI/CD (Continuous Integration/Continuous Deployment) environment and doing "rolling recoveries" of systems "as you go" which is the case in a DevOps oriented system.

Many of these backup utilities are not necessarily best suited for a CI/CD environment because of the continuous changes taking place on both the development and production environment.

I'm looking at programs like Borg
and Git based backups like Bup
These programs would allow for many incremental backups and finer granularity of backups and work well with CD tools like Ansible.  Even Ansible has a couple of way to make backups.  However, there is no clean way to recover the most recent backup in an automated way.

Relying on some form of caching or mirroring of your content on the cluster and hope it's not corrupted is not a way to go :(

The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary.
The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles.
Currently I use a combination of scp and tar into an archive directory sorted by date and machine.  I like to find something better if possible.

I would like to know what are the best backup and recovery tools used in the CI/CD environment?  and an example of how you would configure it?

I don't expect a "magic bullet" just some possible solutions.

The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary. The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles. Currently I use a combination of scp and tar into an archive directory sorted by date and machine. I like to find something better if possible.

Config files, containers and VMs should be backed up and versioned by a tool like git or SVN if you are doing configuration management properly.. Backing up config files and versioning them is the whole point of configuration management tools like, salt, puppet, ansible, and chef.

As you duly note however, dynamic content such as database data and similarly dynamic data is a more challenging issue. These however, are outside of the scope of configuration management, CI, and CD principles. These principles purposefully remain silent on dealing with these issues: because they have no new answers for these problems. Instead, you should use the many battle-tested techniques and strategies for dealing with this kind of data instead of expecting CI/CD/CM tools to solve problems that they were never intended to address.

I think from a DevOps POV that you are approaching this from slightly the wrong angle. DevOps thinking for example is that there is no need to backup a configuration file; the configuration file is written by a configuration management tool such as Chef and can be recreated at any time.

The database does need to be backed up obviously as that is the truly stateful component of the whole system; you don't mention which you are using but head on over to DBA.SE and someone (perhaps me) can give you some specific pointers on that side of things.

For the uploaded content, if this is outside the database, I'd recommend some filesystem-snapshot based approach, it's easy to do and low overhead.


Why `docker stack` needs Docker to run in swarm mode? What are the implications when working on a single node? I am experimenting with docker stack as a replacement to docker-compose. But I had the following message on my first deploy attempt:

sudo docker stack deploy --compose-file stack/app-stack.yml app-stack

this node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again

Once issued the docker swarm init command, docker stack deploy works as expected.

Why it is mandatory to switch to swarm mode to run docker stack? And most important, what are the implications of switching to swarm mode when working on a single node?

Why it is mandatory to switch to swarm mode to run docker stack?

A stack and service are swarm mode objects by their very definition. The stack is a collection of services, and each service is a definition of a target state. Swarm mode takes those target states and manages containers to achieve that target state.

  And most important, what are the implications of switching to swarm mode when working on a single node?

Pretty minimal from the user perspective. There's another few ports opened and listening. If you try to directly manage containers (stop, rm, restart) that are managed by a service, you'll find that swarm mode will take steps to get you back to the service definition, which confuses some that keep finding new containers recreated when they delete all running containers directly. You just need to run the docker service commands to manage them instead.

docker stack api is introduced from docker version 1.12.0 and above.
the api is enabled in swarm mode.

to enable the docker stack api, run below command
docker swarm init

when you run the above command, your host would become single master docker swarm cluster.
you should be able to use docker stack and docker service and other new api's.

you can also add new hosts to the swarm cluster as a worker or manager


How can I convert a production Linux server into a docker Image or container? I've a production Ubuntu that contains a huge number of software and dependencies, I want to 'clone' that VPS into a Docker image to be able to replicate the same environment and distribute them in different servers.

Is that possible without any hack?
If yes, what are the big lines to follow?

Is that possible without any hack?

Yes, it is possible. While Docker is often used for microservice-oriented images (i.e., one single service per image), nothing keeps you from converting a monolithic old application to Docker by building a large image, and running arbitrarily many processes inside.

While it is true that a Docker image is not a standalone VM, it still contains everything an equivalent VM would contain - excluding the kernel. I.e., all OS libraries (libc, everything else...) are in there anyways.

  If yes, what are the big lines to follow?

A Dockerfile is just a funky version of a plain old shell script which you could ordinarily use to provision your software with all its dependencies. That is, you chain a lot of commands, which are executed during the build process, and whatever ends up in the file system at the end gets packed up into an image.

So the general route is:

Pick a base image. Depending on your needs and security requirements, you can pick your favourite distribution from Dockerhub. At first, you can pick any Debian, Ubuntu or whatever distribution you prefer and are used to. See below for further considerations.
Those base images usually are pretty small. Hence, your next step is to use RUN apt install... (or whatever package manager your base image uses) to add any standard packages you like.
Finally, you have to install your own software somehow. If you follow the usual approach, you would prepare any binaries and other files you need in the host-side directory alongside your Dockerfile, and then ADD them in.
The last step is to (optionally) create a ENTRYPOINT, i.e., a command that is run when the image is started up, as well as EXPOSE routing information, run as non-root USER or set ENVironment variables.

These are the big lines. If you just do it naively, you will end up with a huge image (gigabytes), and create a new huge image everytime you rebuild, wich can pose serious problems. The next steps are thus:

Switch the base image to a smaller one. alpine is good.
For each RUNcall, make sure there are no temporary/unneccessary files left afterwards. For example, apt install leaves you with a lot of cache junk in /var (same for the other package managers). Due to the way Docker works with its onion-style layered file systems, you have to get rid of the junk within the same RUN command, you cannot clean up in a second RUN. If you google, you will find plenty of examples.
You probably pulled in too many packages with your package manager, fix that.
For installing your own software, the same applies. Avoid having temporary .tar archives lying around which you don't need anymore.

These steps should significantly reduce your image size already. Depending on what you are actually doing, 500'ish MB is a good target to avoid over-optimizing at this stage.

The final step is to optimize everything:

Make sure that you do not have to rebuild every single intermediate image each time, by ordering the lines in the Dockerfile appropriately. You can get away with large images if the "largeness" happens in the first few image layers, if those change seldom, and can be re-used by many final images.
For an intermediate image to be re-used, it must be bitwise identical to another one. So be sure to fix all variable information (like version numbers etc.) as best you can.
Obviously, if you get the hang of all of this, and get your complex, many-part application running inside a single Docker image, then it's time to split it apart across multiple images (rinse, repeat). Anything that communicates by TCP/IP should be an easy choice to do so.

The whole reason for Docker is to not need an entire operating system underneath. If you want an entire operating system, just take a VM and clone it and distribute that.


Can large scale services use a public cloud managed database service? Theoretical question:
Is it possible for a very large scale service to use a major public cloud managed database service?
For example, a service with a scale of Netflix and a service like Amazon Aurora.
If not, which solutions are available for very larges scale services?

Yes, it is certainly possible, but it entirely depends on the architectural design of the entire system. Separating application services into manageable components (i.e. microservices) is one approach to this. While Netflix as a whole is one application in terms of the user experience, it is comprised of many applications/services that provide a small fraction of the functionality needed to run Netflix as a whole.
In terms of a database, the same principal can be applied. Netflix doesn't manage one database for the entire application, each component utilizes it's own scalable database service.

My conclusion after additional research and Preston Martin answer:
It depends.
If the workloads doesn't require more resources than the maximum supported by the service:
maximum horizonal and vertical (instance type) scale of the servers.
If it does, the data should be distributed across multiple managed database clusters.
If that doesn't meet the requirements, this solution should help:
Vitess


How to execute an ansible playbook using a jenkins pipeline script? We  developed declarative pipeline code to execute an Ansible playbook.
We tested the playbook from CLI and it is working properly.
My playbook actually performs some shell commands on remote server.

SSH configuration between two servers has been done: I have
generated an ssh key and copied it to the remote server, and I am able to ping the server.

The Error msg:

ansible-playbook /var/lib/jenkins/workspace/test1/playbook2.yml -i
/var/lib/jenkins/workspace/test1/inventory.ini -b --become-user root
[WARNING] Ansible is in a world writable directory
(/var/lib/jenkins/workspace/test1), ignoring it as an ansible.cfg source

fatal: [17.14.69.21]: UNREACHABLE! => {"changed": false, "msg": "Failed to
connect to the host via ssh: Permission denied (publickey,gssapi-
keyex,gssapi-with-mic,password).\r\n", "unreachable": true}
[17.14.69.21]:
FATAL: command execution failed
hudson.AbortException: Ansible playbook execution failed.

I have given sudo access to Jenkins user and Jenkins is running as Jenkins user.

If you have Jenkins user public key, on your target hosts authorized keys file, for the user you want to login as, ssh connection authorization should work.

This is an example of Jenkinsfile step.

ansiblePlaybook colorized: true, installation: 'Ansible', inventory: 'inventory', playbook: 'playbook.yml',tags: "" , skippedTags: "", extras: "--extra-vars ' hst=targetHost "

This is Ansible Inventory example. You can define the args at the host level as well.

[targetHost]
17.14.69.21 ansible_ssh_user=sshuser

[all:vars]
ansible_connection=ssh
ansible_ssh_common_args='-o StrictHostKeyChecking=no -oHostKeyAlgorithms=+ssh-dss'

--become-user root

I think you are doing ssh connection between two servers as a root user. I would like to know why. I would never do this.

Assuming those servers are Unix/Linux . Since there is no details how you test with CLI.I am also curious about it.

This might help.
Enable root login over SSH

  Command line options
  --ask-become-pass, -K
      ask for privilege escalation password; does not imply become will be used. Note that this password will be used for all hosts.
  --become, -b    run operations with become (no password implied)
  --become-method=BECOME_METHOD
      privilege escalation method to use (default=sudo), valid choices: [ sudo | su | pbrun | pfexec | doas | dzdo | ksu | runas | machinectl ]
  --become-user=BECOME_USER
      run operations as this user (default=root), does not imply –become/-b

Understanding Privilege Escalation

I think you just want it working :) Ansible documentation is short and brief.

  Working with Inventory
  ansible_ssh_pass
      The ssh password to use (never store this variable in plain text; always use a vault. See Variables and Vaults)
  ansible_ssh_private_key_file
      Private key file used by ssh. Useful if using multiple keys and you don’t want to use SSH agent.
  ansible_ssh_common_args
      This setting is always appended to the default command line for sftp, scp, and ssh. Useful to configure a ProxyCommand for a certain host (or group).

Working with Inventory


What are the signs of an understaffed DevOps team? What are the typical signs and signals of a DevOps team being understaffed? How would you justify/explain a request for a new addition to a team?

I would love to keep the question generic, but here is some additional information:

We currently have 2 DevOps specialists working together as a team, but the demands and the amount and complexity of products are growing. We are thinking to request a new addition to the team, but having some difficulties explaining and proving why it would be a good idea.

There are four main reasons why you can feel your team is understaffed:

Poor organization and planning of work
Doing work someone else should be doing
Doing work that should not be done at all
Being actually understaffed

Start with a review of the first three points. Read the Phoenix Project on ideas how to do the first. Ask yourself for every task you help anyone with if it should be done at all and if it is you that should be doing the task or if you should simply enable whoever needs it done to do it themselves. This will give you some documentation on why all the work you do is necessary.

Next review the four types of work mentioned in the Phoenix project:

Business Projects - what you do for other teams in the organization
Internal Projects - what you do to make your work easier in future
Scheduled Maintenance - what you do to keep the lights on
Unplanned Interrupts - what you do because something went wrong

If the work of your team is sustainable, you will spend roughly the same amount of time on each of the four. If the unplanned work starts to creep up close to 50% of your time, it is a sign you are definitely understaffed.

You should be able to hire to stay about one person ahead of the unplanned work reaching 25% of your time, otherwise, one person leaving will send your entire team into a tailspin you might never recover from. Overprovisioning of people and technology have same reasons and benefits.

Grouping of tasks

An approach we've used in the past in similar situations is to organize the work of a team in 4 major groups of tasks, and allocate the equivalent of 2 FTE (Full Time Equivalents) to (try to) complete those tasks. In our case it was related to running an SCM helpdesk in a mainframe environment, with about 300 developers asking for all sorts of help / interventions from those 2 FTEs. The groups of tasks are organized in 4 possible priorities:

Tasks of Priority 1 and 2 must be completed (no excuses / negotiations possible)
Priority 3 tasks are to be completed "as soon as time permits".
Priority 4 tasks are to be completed "if time permits".

Read on for more details about the kind of tasks in each of those 4 groups ...

Task descriptions

Priority 1 - Operate the helpdesk

By experts who are easily accessible and always available.
Via phone, eMail or ticketing system during business hours.
Compliant to predefined SLAs.
ITIL-based registration of all helpdesk calls with periodic reporting of all calls.
Apply emergency customizations (work arounds) for critical issues.

Priority 2 - Watch duty services

24h/day, 7d/week on-call availability.
Compliant to predefined SLAs.
Reporting of all watch duty calls.
Management escalation where needed.

Priority 3 - Routine maintenance

Administration.
Application boarding.
Housekeeping.
Performance enhancements.
Space management.
Tuning of resource consumption.
Suggest enhancements for customisations to reduce the number of helpdesk calls and/or watch duty interventions.

Priority 4 - Fixes and enhancements

Create and maintain user documentation.
QA testing of new customisations.
Develop and implement enhancement requests.
Participate in DRP-testing.

Evaluation

If you're using an approach as described above, various things may (will!) start to happen:

If the team is understaffed, probably most of the time will go to priority 1 and 2 tasks, while it may take a while to get the priority 3 tasks ... and priority 4 tasks might suffer starvation (there appears to never be time for those tasks).
The more there is (becomes) time available to "invest" in priority 4 tasks,  the more time needed for priority 1 and 2 tasks will be reduced, so that even more time (of the available budget of 2 FTEs) can be "invested" in priority 4 tasks.
You'll be amazed to see how, after a while, the number of priority 1 and 2 tasks will go down. And if you do adequate reporting about those tasks, that's something management loves to hear. In our case that number went down from about 300/month, to below 100/month ...
If however the 2 FTEs appear to never (or hardly) have time for priority 4 tasks, then you have a perfect explanation and proof for your management ... that you're understaffed.


Docker container host machine communication Hi i’m trying to communicate with mongodb in the host from a flask container…
So using “docker.host.internal” on the URI didn’t work (host machine is Ubuntu)
Any suggestions ?? To solve this

You can run your flask container using the host networking mode. This way docker won't isolate the container and will run it the same network namespace as the host, and you container will be able to discover your database using localhost as mongodb hostname
docker run --network host <flask-image>

You can add an alias IP address to the loopback interface on the host to get a consistent address to use from your containers.
Any private address you won't see on the local or container networks, something like 10.8.8.8
Edit /etc/network/interfaces.d/lo for a permanent address after reboot
auto lo
iface lo inet loopback
iface lo inet static
  address 10.8.8.8/32

or to temporarily test
ip address add 10.8.8.8/32 dev lo


Is there an ansible module that check whether a port is not in LISTEN state? The Ansible module index has been consulted, but no module was found that checks whether a port is available? Is there a module that supports this?

What does available mean?

Available means that a port, e.g. 80 is not in LISTEN state. The following indicates that port 111 is unavailable.

user@localhost ~ $ ss -nat
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port
LISTEN     0      128       *:111                   *:*

What is the aim of this question?

The aim of this question is to find a way to stop an Ansible run when port 80 is already allocated by another process than certbot. Certbot requires port 80 to be available, otherwise the cert cannot be renewed. The issue can be found here.

I think you're looking for the wait_for module. It will allow you to check and make an action when a certain port is available and more.

Letsencrypt is long-time renamed to certbot. You can automate certificate renewal process by configuring a host which serves a domain(s) by installing certbot and configuring web server for handling a directory with auth tokens as /.well-known/acme-challenge/ for domains under Let's Encrypt-signed certs, then using Ansible playbook for manual renewal process or run the same command by cron.


What are best practices for using Ansible Vault on public CI's and Source Control Repositories like BitBucket? Introduction

On a private CI's and Source Control Repositories like Gitlab-ce it is possible to copy an ~/.vault_pass.txt to the server and let it use by the CI to decrypt files using Ansible.

Problem

On public CI's and Source Control Repositories like Bitbucket it is not possible to copy a ~/.vault_pass.txt to the CI server it self.

Discussion

In Bitbucket it is possible to define encrypted variables, but when this file is checked the only VAULT related variables are:

ANSIBLE_ASK_VAULT_PASS
ANSIBLE_VAULT_PASSWORD_FILE

These variables are not an option to solve the issue as when the ANSIBLE_ASK_VAULT_PASS is set ansible-vault still prompts:

user@host $
Vault password:

When the same password is entered it can open the encrypted file, but the aim is to open the file without needing a file or entering a password in a prompt.

Another attempt to solve the issue was running export ANSIBLE_ASK_VAULT_PASS=<ansible-vault-password>, but the interactive mode persists.

Another option is export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass.txt, but then this file needs to be pushed to the repository, but source Control repositories should not contain passwords.

--vault-password-file can instead point to an executable script that writes to stdout. This little-known feature should solve your problem.

First, write a simple, executable script that prints a system environment variable, and check that into your source control. Then, use Bitbucket's encrypted variable feature to set that environment variable to your ansible-vault secret. Finally, execute like so:

ansible-playbook site.yml --vault-password-file ./mypass.sh.

References:

http://docs.ansible.com/ansible/playbooks_vault.html#running-a-playbook-with-vault
https://groups.google.com/forum/#!topic/ansible-devel/1vFc3y6Ogto

You could have a test set up that does not run production, and load different files for those.

Create a host_vars/localhost/vault that only works for the local test installation.

That way you could use a open vault password that works only for that local host vault.


What is the advantage of using Prometheus over ingesting metrics using logs? I want to setup monitoring on our environment. I am wondering on what are the advantages of using something like Prometheus for making metrics vs. just using something like the ELK stack for creating similar metrics using logs?

Efficiency

If you don't need the file it saves space and trouble to leave it out.  Logs require rotation and retention policies which are nice to skip.

Reliability

Skipping the file avoids issues with the file that could caused lost metrics like a full disk.

Latency

Skipping the file means that you remove latency from disk I/O.  For most metrics you don't care if they show up a few seconds late, but if you're trying to build things to respond to some metric quickly.  You might want to put it on Kafka also.

The basic difference between prometheus metrics And logs is that prometheus is a timeseries db  while logs - in General sense - are event driven.

Thus using prometheus is suitable for monitoring the whole lifecycle of your endpoints, making aggregations on results And generally evaluating the behaviour of your target before something occurs.

Logs are more suitable if you want reportin when something particular happens, in an order of events And generally tracking down incidents after they've already happened.


How Do I Log Task Output To A File? One of my ansible tasks imports an Oracle database using impdp.

This generates a lot of output to the console so I have set no_log: True.

However, when this fails I want to see the log!

How can I make this particular task log to a file and not to the console?

[Converting my comment to an answer]

One way to do it would be to write the logs to some external file, and then having a task after it which makes use of failed_when condition, and remove the log file, if the previous task was successful.

Something like this should help you.

 - name: Run Py script
      command: <>.py  > <>.log
      become: yes
      register: PyScript
      ignore_errors: True

    - name: PyScript on success
      command: rm <>.log
      when: PyScript|succeeded

Note: This might not be the best way to handle your problem. But, this was a hack which helped me do my logging and monitoring.

Tee will be a very simple tool for logging, you can refer the following command.

eric@eric-MacBookPro:~$ ansible -m ping all | tee > /tmp/ansible.log
eric@eric-MacBookPro:~$ cat /tmp/ansible.log
localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"
}


Is it possible to build Jenkins jobs without using build numbers? I have a job which is running very frequently, so I'd like to avoid high number of builds. Is there any way to drop job build numbers, so the job is constantly ran, but without high numbers of builds?

It's not possible to avoid setting the BUILD_NUMBER environment variable, but you can avoid using it. Try using the Jenkins plugin Build Name Setter Plugin. It adds an option in your build configuration to set the build name to whatever you want, and you can ignore the BUILD_NUMBER environment variable this way.

Consider 1.250.5 versus 1.250.${BUILD_NUMBER}. The former will be constant no matter how many times you build and the latter will increment by one every time you build. It's also possible to use other macro values in between ${} to make more interesting dynamic build numbers.

No, but you can just ignore it which you would be doing anyway if it wasn't there.