119647-development-process-question

== Content ==
			
Hello All,
 
I am sure some others have the same question as I, so I might as well ask (don't flame because I, like you, have followed this game for a long time and understand programming takes time)
 
 
Anyhow, looking over the 12/9 patch notes, i cant help but wonder, 
Why does it take so long to get such little fixes out?
 
I was reading through the patch feedback and one of the questions struck me. Why does it take 5 weeks to fix things that were supposed to be corrected in drop 3?
 
"Programming is hard, tedious work that can potentially take a long time to resolve." Okay, I get that part.
 
"some problems are hard to duplicate" well, you are programmers working on software you have had 10 years to get used to, what is the problem?
 
"we have to get approval first before putting it into patch" well dont say its going to be fixed then take 5 weeks to fix it?
 
Sorry, but i am a customer, and i have paid already hmmm.. i think like 130$ ish for your game. Can we get some more 'speedy' results please?
 
this has been discussed so many times.....	
----
			
Game Development is a process that is hard to understand until you are in it.  I thought the same way before getting into the industry.  Basically working with multiple teams trying to prioritize different initiatives and then incorporating the scheduling of the entire studio around those things, it can get pretty complicated.  It is quite a process to get items fixed, tested, LOC'd, pushed to PTR, tested again, then pushed to Live. Now I am not making excuses for us.  I think we as a studio, myself included, can do better at communicating certain things to our player base.  
 
I understand your frustration, just know that we are actively listening to your feedback on a daily basis and that we are doing our best to get prioritized fixes into the game as quickly as possible.	
----
			They also have to take pains to not introduce new bugs.	
----
			
{| class="blockquote"
 well dont say its going to be fixed then take 5 weeks to fix it?
----
			
{| class="blockquote"
They also have to take pains to not introduce new bugs.
----
			
I really don't find it that LOL-worthy that they went from a once a month to quarterly release schedule to improve bug hunting and then released arguably the most damaged product to date.
 
Though in some cases deliberate idiocy (level locking mining nodes, soulbinding thousands of purple grade items without warning, recoloring harvest node icons the same color as hostile mobs) almost overshadow the dozens of game crashing bugs they introduced.	
----
			
{| class="blockquote"
I really don't find it that LOL-worthy that they went from a once a month to quarterly release schedule to improve bug hunting and then released arguably the most damaged product to date.
 
----
			
Sounds like a development company with standard development processes, that always/mostly leads to buggy, unmaintainable code, delivered to late, aso. As one can find out from the internet, it´s really that way.
 
I would suggest Carbine to look at e.g. http://www.versionone.com/agile-101/agile-software-development-benefits/
 
Quote:
In particular, agile development accelerates the delivery of initial business value, and through a process of continuous planning and feedback, is able to ensure that value is continuing to be maximized throughout the development process.  As a result of this iterative planning and feedback loop, teams are able to continuously align the delivered software with desired business needs, easily adapting to changing requirements throughout the process.  By measuring and evaluating status based on the undeniable truth of working, testing software, much more accurate visibility into the actual progress of projects is available.  Finally, as a result of following an agile process, at the conclusion of a project is a software system that much better addresses the business and customer needs.
 
Although, its not a quick solution of the problem, but for the future !	
----
			I would be surprised if they're not using Agile methodology.OH and to everyone saying that Drop3 was hardly damaged... Drop 3 introduced the worst problems out of all 3 Drops that took weeks to fix. 	
----
			
{| class="blockquote"
I would be surprised if they're not using Agile methodology.
 
OH and to everyone saying that Drop3 was hardly damaged... Drop 3 introduced the worst problems out of all 3 Drops that took weeks to fix.
 
----
			
{| class="blockquote"
Wasn't sabotage totally broken though and that started all the changing of how they would introduce things
----
			
{| class="blockquote"
I would be surprised if they're not using Agile methodology.
 
 
 
----
			
{| class="blockquote"
it didn't break anywhere near as bad as Drop 3. In Drop 3 everything broke. LAS broke, ability to change abilities or action bars broke, housing harvesting plots broke, NPCs that take you to zones broke... IT was near unplayable for days and still very annoying for a week or two. 
----
			
{| class="blockquote"
You will be suprises. Simple google for carbine jobs and read, what they are asking for. e.g.: "Desire/ability to work in a team environment"
Seems, they never heared the word "agile".
----
			
{| class="blockquote"
that's required for agile as well. I'm in uni and year 1 we did waterfall, year 2 we did agile. Both have to do with working in a team based environment. The only difference is that agile has daily meetings, daily scheduling (and re-schedulling the next day depending on what you did the previous day), involving testers from as early in planning as possible and going trough a constant feedback -> concept loop. 
----
			
{| class="blockquote"
If you only implemented those few elements, then you did not realy accomplish agile development. Agile development is much more than doing some dailies. This is one of the major misconceptions, where companies switch to "agile" and wonder, why nothing gets better.
Perhaps I better should have talked of "Agile Productmanagement"
 
Perhaps you read here: http://www.romanpichler.com/
----
			
Remember that when they got started, Agile wasn't the hot thing it is today. It takes time to switch methodologies.
 
And even today, most people who try Agile do it wrong.
 
Of my last four companies only one did Agile really well, and two of the companies had the exact same team, but different corporate culture. What flourished in one environment only worked moderately in the other.
 
Who knows what struggle they are dealing with in a large, complex system developed over 10 years? The code base is likely complex and difficult to work with (with so many different teams/coders in the mix).
 
For the developers out there, think about what you were doing 10 years ago. Ideas like IOC that I wouldn't live without today were unheard of then (not saying they are using IOC in game development- I don't know what ideas/tech is used- but the industry changes rapidly is my point).	
----
			
Agile's just an excuse to not have deadlines!!  Everyone always says it's not but in reality you get requirements slipping in left right and center and if it's not managed correctly (90% of the time it's not really) you end up with slippage.
 
You also have the fact there are a number of teams working on things and as we were told in the Esper forum some issues we are having are actually an issue due to another teams work, not the developer of the class as such because it's using functionality provided from another team. (Could be a curve ball!! :P)
 
That's why changes have unforeseen problems and you end up with people saying "yep we've fixed it"...they have fixed it but it can't go in until the other team says so in case they are making changes that then break it further etc.  There are dependencies there that aren't working smoothly from the outside looking in.
 
Then you have the whole issue of source control and how changes are made and when they are merged back in etc, if you have two changes that work fine on there own then you put them in having not tested it fully they can clash and break things.	
----
			
{| class="blockquote"
the dozens of game crashing bugs they introduced.
----
			
{| class="blockquote"
Who knows what struggle they are dealing with in a large, complex system developed over 10 years? The code base is likely complex and difficult to work with (with so many different teams/coders in the mix).
----
			
{| class="blockquote"
Agile's just an excuse to not have deadlines!!  Everyone always says it's not but in reality you get requirements slipping in left right and center and if it's not managed correctly (90% of the time it's not really) you end up with slippage.
 
You also have the fact there are a number of teams working on things and as we were told in the Esper forum some issues we are having are actually an issue due to another teams work, not the developer of the class as such because it's using functionality provided from another team. (Could be a curve ball!! :P)
 
That's why changes have unforeseen problems and you end up with people saying "yep we've fixed it"...they have fixed it but it can't go in until the other team says so in case they are making changes that then break it further etc.  There are dependencies there that aren't working smoothly from the outside looking in.
 
Then you have the whole issue of source control and how changes are made and when they are merged back in etc, if you have two changes that work fine on there own then you put them in having not tested it fully they can clash and break things.
----
			
As above mentioned, most "agile" tries are done very bad. Often its just an alibi of company leadership. They pretend they have reacted on bad development performance. In fact, they only permit to implement some "things" like "dailies" and other minor changes, but important changes are not accomplished. You can read examples above.
To handle large projects with complex code, its necessary to implement further develpoment processes. Short sprint periods with the need to deliver a runnning system at its end, needs e.g. intensiv testing (unit-testing, acceptance-testing), close coworking of developers and QA stuff. Also tools for cross-team coordination are available.Its not easy to move a "standard" development team from usually unstructered methods ("chief calls, developers code"). Especially problematical is to bring legacy code under test.
But all problems can be adressed with the right methods and improve the product management and development processes and its worth to go that way.
Its a matter of good will, problem comprehension and knowhow. Sadly many company leads have large gaps within that.	
----
			
Hi All,I'm one of the Agile Scrum proponents over here at Carbine.  
 
I've had hands on experience with Traditional Project Management (TPM / Waterfall), Agile Project Management (Scrum), Lean Product Management (Six Sigma Lean), and eXtreme Software Development methodologies as they apply to both software development and manufacturing.
 
Please feel free to reply with any questions you may have about our development or operations processes and I will try my best to provide clarification and explanations when I have bandwith.
 
Cheers	
----
			
Hi, Virtue,Thanks to offering the possibilty to ask questions about that. I know, it may be difficult for you togive in depth looks at some internal company processes.Well, the starting point of the discussion was, "Why does it take so long to get such little fixes out?".And behind this, of course always the same questions about "why does development goes so slow, has so many bugs, etc."and especially "why are deadlines broken that much ?"From my experience as a product manager and software developer, I know that all those problems are the samein most companies. And therefore one searches for reasons and solutions.Ordinarily software is produced in the "standard way", solution may be switching to "agile" processes.As I can see from company infos in the internet, I would guess, that Carbine doesn't use agile product management.Your words "proponent of scrum" seems to approve that.
 
Am I right ?If not, can you describe, what kind of solution/s) Carbine tries, to overcome the production problems ?
 
Cheers	
----
			
{| class="blockquote"
As an Agile Coach....that's bullshit.
Just cause you're agile doesn't mean you get to shove stuff to the side.
Agile means being flexible enough to actually meet deadlines and focus on what's important....
----
			
{| class="blockquote"
Yes if it's done properly and as I said in reality a lot of the time it's not.  It was tongue in cheek anyway about moving deadlines.
 
Certainly when people are trying to adopt it when they don't have the foundations set for it.
----
			
{| class="blockquote"
You'd be surprised how much people sabotage their own work and colleagues just out of fear for changing something.
----
			Yep, same old story with a lot of places, the old big wigs and all that and you certainly see these things happening all too often.	
----
			
{| class="blockquote"
Hi, Virtue,
Thanks to offering the possibilty to ask questions about that. I know, it may be difficult for you to
give in depth looks at some internal company processes.
Well, the starting point of the discussion was, "Why does it take so long to get such little fixes out?".
And behind this, of course always the same questions about "why does development goes so slow, has so many bugs, etc."
and especially "why are deadlines broken that much ?"
From my experience as a product manager and software developer, I know that all those problems are the same
in most companies. And therefore one searches for reasons and solutions.
Ordinarily software is produced in the "standard way", solution may be switching to "agile" processes.
As I can see from company infos in the internet, I would guess, that Carbine doesn't use agile product management.
Your words "proponent of scrum" seems to approve that.
 
Am I right ?
If not, can you describe, what kind of solution/s) Carbine tries, to overcome the production problems ?
 
Cheers
----
			
Hi Virtue,
 
I am really suprised by your detailed answer and descriptions. Thanks very much !
 
When you and Carbine consequently walk this way, I am sure, things will run better in the future.
Yes I know, it takes some time to tranform a complete company/development team(s) into new structures, and that it slows down velocity in the beginning. Its not easy and one must fight through this situation. But at least, it will be worth it. And good to hear, that the necessity to change production processes, eg. by using agile methods, has found its way into heads,
 
I hope, you will have the time and resources to complete this process, and we will see those improvements in the game(s) you provide us with.
 
Thanks, greetings and good luck for all staff.	
----
			
Hi Virtue,
 
may be, it goes to far. Regarding the test-process:
There is a long queue of systems (Development | Staging | PTR Unlocked | PTR Mirror | Live Mirror).
 
Do all tests run through all of this systems ?
 
Can you say something about counts, at which point/system most of the tests fail, and change/feature goes back to development ?
 
How much of your tests are automated or manual ?
 
Cheers	
----
			
{| class="blockquote"
Hello All,
 
I am sure some others have the same question as I, so I might as well ask (don't flame because I, like you, have followed this game for a long time and understand programming takes time)
 
 
Anyhow, looking over the 12/9 patch notes, i cant help but wonder,
 
Why does it take so long to get such little fixes out?
 
I was reading through the patch feedback and one of the questions struck me. Why does it take 5 weeks to fix things that were supposed to be corrected in drop 3?
 
"Programming is hard, tedious work that can potentially take a long time to resolve." Okay, I get that part.
 
"some problems are hard to duplicate" well, you are programmers working on software you have had 10 years to get used to, what is the problem?
 
"we have to get approval first before putting it into patch" well dont say its going to be fixed then take 5 weeks to fix it?
 
Sorry, but i am a customer, and i have paid already hmmm.. i think like 130$ ish for your game. Can we get some more 'speedy' results please?
 
this has been discussed so many times.....
----
			
{| class="blockquote"
Hi Lusalot, 
 
I'm going to break down your questions and answer them as best as I can.
 
Why does it take so long to get little fixes out?	Our internal environments in their current capacity look like the following:	Development | Staging | PTR Unlocked | PTR Mirror | Live Mirror 	
Our external public facing environments look like the following:	PTR | Live	
Any feature development or bug fix must be checked into an environment in order for the change to take effect.  A "build" of the game server and clients must then take effect for our internal devs to verify that the change was made. In the case of a bug on Live, we have to check the fix into Live Mirror and all of those environments that fall below it.  We have tools that allow us to reduce the overhead of checking in across all environments, but we still have to assess every live bug and determine from a change management perspective how risky the fix is and the scope of QA time required to test it.	Based on an objective severity rating and taking into account subjective opinions from our community and developers, we triage out issues to determine which environments they should be fixed in and when their intended deployment will occur.	Once an issue is fixed and a build has been provided, it passes to our QA and Release Management team.  The goal of these teams are to perform smoke tests, full regression testing, and BVTs on the builds that are created.  If a build passes these tests and all of the necessary certifications have occurred with the Release Management team, a time is scheduled for deployment to Live, patch notes are generated, and the build is pushed to the public PTR or Live environment.	Between bug fix, triage, build, QA, and release there are a lot of moving pieces and dozens of teams that need to verify, approve, and release the content, resulting in sometimes delayed or longer than liked release windows. 		Why does development take so long and why are there bugs in content?	Development of new features requires a lot of studio buy in and resource commitment to begin.  The scope of features is often huge and they must proceed through four distinct phases:	Pre-Production (the early design documents are drafted and discussions about the philosophy or goal of the feature are had)		Prototype (an early playable working prototype is created)		Production (the full commitment of asset creation, engineering efforts, etc. are done for the feature)		Post-Production / Polish (this is very much the polish phase of a feature, where remaining UX bugs are triaged, fixed, etc.)	
Once these phases are complete, the feature has to make its way through rigorous QA and Release Management processes.  We have a certification process where we attempt to identify and fix many of the critical issues associated with a feature.  Once the QA and Release certification process is complete, we are ready to deploy the feature in our quarterly patches.  	In our current capacity, we run continuous integration against multiple development environments with multiple development builds per week.  Moving forward, in order to better embrace Agile practices and to reduce the risk associated with large scale CI on new features, we will be moving to a code trunk.		Why are deadlines broken?	Historically we did pretty well with hitting our release cadences, this can be noted for Drop 1 and Drop 2 which both went out as planned.  Unfortunately, as we found and so did players, the quality of our release were not high enough.  Internalizing player feedback we made the decision to move away from monthly releases in order to better provide the end to end testing required for a quality product and to empower all of our teams to fix and identify major issues in their content.	In our current capacity, I am very optimistic that our future development velocity will increase, unfortunately, as many teams know, when changing processes a huge reduction in velocity occurs.  We are at this stage currently and I am hopeful that by early next year we will see studio wide increases as we better adapt to the changes we have implemented. 	
Does Carbine use Agile Scrum?	Yes and no.  We approach project management with a tool set.  We look at teams and we see that Agile Scrum works great for some of them and that Kanban or even Waterfall works better for others.  I see ourselves adopting more and more Agile methodologies as we move into the New Year and I am hopeful that a lot of the principles that are core to Agile have a grassroots growth at Carbine.  	A great example of this is a blending of eXtreme Software Development and Scrum on some of our Engineering teams.  They firmly enforce paired programming and code reviews and in addition to this are able to voice blocker issues or discuss cross feature/disciplinary issues during their daily standups.  	In conclusion, I think we are making huge strides towards the Agile Manifesto:		Individuals and interactions over Processes and tools	
	Working software over Comprehensive documentation			Customer collaboration over Contract negotiation			Responding to change over Following a plan				More importantly, I think we are making strides towards the core Agile Principles as well:		Customer satisfaction by rapid delivery of useful software	
	Welcome changing requirements, even late in development			Working software is delivered frequently (weeks rather than months)			Close, daily cooperation between business people and developers			Projects are built around motivated individuals, who should be trusted			Face-to-face conversation is the best form of communication (co-location)			Working software is the principal measure of progress			Sustainable development, able to maintain a constant pace			Continuous attention to technical excellence and good design			Simplicity—the art of maximizing the amount of work not done—is essential			Self-organizing teams			Regular adaptation to changing circumstances			The truth is that we aren't there yet, but we are committed in making a huge effort in trying to get there.  While all of these changes aren't easy to implement across an entire organization, it's the ability to iterate and apply the toolsets we have at our disposals that we will continue to rely on to increase efficiency and value for our players. 	
 
Cheers
----
			
{| class="blockquote"
it didn't break anywhere near as bad as Drop 3. In Drop 3 everything broke. LAS broke, ability to change abilities or action bars broke, housing harvesting plots broke, NPCs that take you to zones broke... IT was near unplayable for days and still very annoying for a week or two. 
----
			
Wow, Virtue...that's a lot of info I never even knew I wanted. Since I know nothing about software development, I'm not gonna comment on the process other than to say good luck with the changes!!
 
As a player, what I can say is that if what we are seeing in Drop 4 is due to these new process....please keep going!!! It's like Christmas over there! So many things in there I didn't even expect. I think I'm more excited about Drop 4 than I was about the game's original release. So, thank you.
And again, good luck. My company has been going through lots of process changes, and I know how frustrating they can be. However, smoother processes will help everyone, in the end. Hang in there!! And thanks for sharing some insight with us :)	
----