Thought Experiment: Evolving a Ticket & Incident Team to 50% Development Time
By: Malcolm Stanley
Tags: Sre, Software Development, Waterfall, Team Development, Agile

Different Types of Work Demand Different Types of Processes, Tools, and People

Scenario

I participated in a scenario-planning session discussing what it would take to enable moving a ticket and incident response team for a complex network to 50% development time.

The idea, more or less, is to migrate the team over time to a more “SRE”-like approach. The team considered for this activity is 100% allocated to ticket and incident response and has little if any development experience. There are no development tools or infrastructure in place. The question posed was how to organize the effort associated with development.

Current work for this team could be characterized as 100% driven by technical debt: software or systems integration defects dominate. Work is accepted via triage of a trouble ticket queue. Tickets in their queue originate in an upstream Operations center that functions as a junction box for numerous integrated vendor systems. The SLA for ticket response is generally pretty immediate, and the ticket flow is full enough to saturate the team on a regular basis.

Members of the ticket and incident response team regularly identify “development candidate” ideas originating in or inspired by their ticket and incident response activities. These development candidates are trapped today in a smartsheet. This smartsheet functions as a rudimentary queue but the reality is that work in this queue is seldom taken up due to ticket pressure.

A strong assumption in the scenario planning is that SRE requires the development of the development candidates to be performed by team members in the ticket flow: developers who do not consume tickets are not SRE in the same way that ticket consumers who do not develop software for a significant percentage of their time are not.

Truly enjoying dinner involves knowing what is on your plate…

Different Food, Different Plate

To help the team managers understand the potential flow of work, we first characterized the different types of software development work that would be available.

They were already very familiar with ticket and incident work: arrives in a queue; has an SLA which requires active queue management to ensure the important stuff is handled first; response to the very urgent items is highly structured with significant rituals and norms to ensure consistency and well-managed expectations. Tools supporting the rituals and norms are already in place.

When we discussed the development candidates arising out of the team, we divided these into two separate work types.

One type of work we expected to see was what we characterized as ‘metric-chasing’. This is where you have a time series measurement and wish to figure out how to make the values you are measuring conform to a goal. Examples would include making response time on a query go down, or decreasing the latency on an autoscaling action. A typical statement here would be “the trend is going toward ‘x’ and I want it to be ‘y’”.

Metric chasing is best accomplished using Lean and Agile methodology: formulate an hypothesis about what will move the number in the desired direction, implement an MVP to see if the hypothesis is correct (Lean) and then quickly iterate and course correct (Agile) as the data provides positive or negative signals in response.

The other type of work we identified is characterized by functional development. This is where a new capability is specified, with a set scope of features to be delivered. This set scope is often accompanied by a demanded delivery date which places such work squarely in the iron triangle. Work of this nature is often managed using a waterfall process, which may encompass the engineers utilizing scrum as a work assignment methodology.

We suggested that the metrics-chasing work and the functional development work be separated, by a Technical Product Owner role, into different queues. This would allow us to apply tools and processes appropriate to each type of work, rather than applying a ‘one size fits all’ approach.

We recommended the metrics work be placed in what we called the “A-queue” (A for ‘Agile’) and that the work management methodology for the A queue be primarily kanban based. The choice of kanban tool was left open, and we specified that additional tools like Wikis needed to be available to support documentation and other activities that kanban tools typically do not support.

We recommended the feature development work go into what we called the “W-queue” (W for ‘Waterfall’) and that an SDLC-supporting platform (like Jira, say) be used to plan the numerous steps required to conform to the complicated test and deployment process in place to ensure that feature updates did not disable critical functionality.

We recommended against mixing these two different types of work in a single queue, with a single set of processes and tools, as we have seen this causes increased complexity in work management with teams, and more difficult expectation management with senior leadership.

Different Tickets, Different Prizes

Diversity in Talents and Choices

The ticket and incident response team has a moderately high level of diversity in the specific technical expertise possessed by its members, in the context of the activities it currently undertakes.

Some members are expert in platform operations, others in networking. Some members have specific expertise in command line scripting, others in automation platforms. Formal application coding experience is not well represented, so hiring focus is being applied to bring that expertise to the team.

The team members have an equally wide range of personalities. Some are more comfortable with decision-tree based activities, others excel at the data-driven investigative methods needed during complex incidents.

It is important to management and to us that the team members have autonomy in choosing the development work they will perform. We discussed the role of the TPO, and how that role could classify work into the proper queues, making sure the definitions of the work requirements and acceptance criteria are clear. That lead to agreement the team members will prioritize the work, which after all originated from them in the first place. The outcome of the grooming described would be two ordered queues of work, with work appropriately sorted by type and urgency, that the team members can draw from.

Part of the experiment here is that we desire the team members to self organize by selecting work, individually or in ad hoc teams, that is important and that interests them. We do not know if this will occur spontaneously, or if nudging will be required, and we are curious if there are learnings in how individual team members make their selections.

One concern we could address: although the queues are ordered, we do not perceive it beneficial to require that work be picked up in queue order, as that would harm the ability to self organize. So all of the work in both queues is fair game for any team member.

If time measurement is not about time… what is it really about?

Measurement

Discussion about how to measure the success of the experiment quickly uncovered a concern about activity tracking. The team today is not time-tracked. The tracking that is performed relates to ticket acceptance and time on ticket. This is used primarily to tune team sizing to ensure SLAs on ticket response are met.

Two questions then arose: how to know what percentage of time is actually being spent on development, against the SRE target of 50%, and how to know what development work was being performed by any individual team member?

As this is a ticket-oriented team, we proposed that each development candidate be tied back to the original incident or trouble ticket that inspired the development candidate. This provides traceability and data around which types of troubles or incidents generate additional development work effort.

We also proposed that we create a new type of ticket that could serve as a ‘development time’ ticket. Each team member would have the ability to assign themselves to such a ticket, and link that ticket to their chosen development candidate, providing end to end traceability. The opening and closing times fore these development time tickets allow a kind of measurement of time spent on the linked development task, without requiring additional individual time tracking overhead to be introduced.

We speculated that with this approach we could generate useful data about work required and performed without sliding into unproductive or superficial productivity metrics.

Nope

Outcome

I’d like to write here that the next blog post will be about the results of this experiment. I’d love to talk about the data we gathered showing what types of incidents generated the most development candidates, or talk about the learning experience of allowing teams to self-organize to accept feature development work in a time-boxed waterfall development candidate queue.

Instead, all I have is the discussion of the scenario and how we put together a plan to address it. The plan we built was not put in place, as so many scenario plans are not. The ticket and incident response team is still responding to tickets and incidents, and doing a great job at it, by the way. Management still wants to move them to 50% development time and reposition them as SRE, but there is no target time frame to do so, meaning the goal may be more of a desire than an actual vision.

Given the project never happened, I am writing this post, to explain our thinking, in case someone else may benefit from it, and possibly run the experiment, or a variation on it, that we did not. I hope the description of the scenario and the way we approached it is useful, and look forward to any discussions people may wish to have about our thinking.