120845-your-patch-process-has-to-change

== Content ==
			
Dear Carbine,
 
Currently your patch process leaves a lot of holes.  There are a number of big issues that have either fallen by the wayside or are "waiting for the next drop" to be fixed.  These fixes need a much quicker "time to market" than what we are currently getting, and none of these issues should be considered drop content to be held up for that patch.  Im confused as to why these patches take so long to into the hands of the people needed to get them out to the player base sooner.  Continuously seeing things like "we have the fix, but no eta on when you will actually get it in", is not reassuring going into a new year.
 
For Wildstar to start shining and earning trust back with players that have left, these issues need to start getting addressed and fixed much quicker.  Waiting for a biweekly or next drop (3 months) is not going to win any points with lost subs.
 
ICC Comm Issue - https://forums.wildstar-online.com/forums/index.php?/topic/118708-iccomm-update/
This issue was reported after drop 3, but wasn't fully realized by the devs until a few weeks later, but no real update has been given except it would be "The next major drop will start to address these issues".  This issue shouldn't be held up for a content patch to address a bug.  Especially one that has been reported since Early November.
 
Support only Archivos weapons - https://forums.wildstar-online.com/forums/index.php?/topic/119127-archivos-cw-5-grs-phageblade/page-2
This problem was reported, in Late November/Early December, and was supposedly supposed to be fixed in the "next biweekly patch" however the december bi-weekly patch came and this was never fixed.
 
Known issues List - https://forums.wildstar-online.com/forums/index.php?/topic/117945-known-issues-mystery-of-the-genesis-prime/
There is still a myriad of issues on the Known Issues List for drop 3 that are labeled as "under investigation" and it seems like Carbine has forgotten about the list.  Can the list either be updated or at least start getting some answers on the list of issues within the last patch.
 
We need some answers, and some updates.  I know there was a long break where people were out of the office for the holidays, but the holidays are over now, some updates would be welcomed.	
----
			Learn how to code and you will understand why it takes so long. Making one change affects multiple systems that then all need changing. Finding the bits of code that need changing takes time, writing new code takes time, implementing and testing the changes on the PTR takes time. There's no MMO magic wand that devs can wave and just make changes happen. Patience is a virtue.	
----
			
{| class="blockquote"
Learn how to code and you will understand why it takes so long. Making one change affects multiple systems that then all need to be changed. Finding the bits of code that need changing takes time, writing new code takes time, implementing and testing the changes on the PTR takes time. There's no MMO magic wand that devs can wave and just make changes happen. Patience is a virtue.
----
			
All three problems have been reported, and Carbine already confirmed they're working on a fix.
I agree, these things should be fixed faster, and potentialy even prevents.
 
The CommChannel not working should have been spotted on PTR by QA.
If not, then there's a serious flaw in the QA process.	
----
			
{| class="blockquote"
Learn how to code and you will understand why it takes so long. Making one change affects multiple systems that then all need changing. Finding the bits of code that need changing takes time, writing new code takes time, implementing and testing the changes on the PTR takes time. There's no MMO magic wand that devs can wave and just make changes happen. Patience is a virtue.
----
			
{| class="blockquote"
Im by no means a programmer, however I do work in an environment that deals with code branches and talking to these guys, yes, i understand the issues. However, you are misunderstanding what im asking here. Im not saying they need to fix problems quicker, Im saying the problem here is a lot of the bugs or issues seem to have a fix somewhere in the pipeline.  A good majority of the time when I read a post by a Dev on an issue I see something like "We have a fix checked in and you should see something in the next drop"
 
My exact issue is with the fact that a lot of these things that are already fixed shouldn't be pushed to the next drop or next biweekly patch.  They need to be QA'ed and pushed out sooner than what we are getting.  Things where the intended functionality doesn't work and is supposedly fixed, shouldn't be made to wait 2 weeks to 3 months for a fix.  The issues I highlighted above should have their fixes pushed out sooner than what they are supposedly scheduled for.  Because regardless of how long they are keeping the drops in QA, its still going to have bugs, and its still going to have issues.  Everytime I see a dev post something along the lines of, "this is fixed, but its in the next biweekly/drop and I don't have an ETA" i cry a little inside because it shouldn't be this hard to get bug fixes out to the player base.
----
			
{| class="blockquote"
Learn how to code and you will understand why it takes so long. Making one change affects multiple systems that then all need changing. Finding the bits of code that need changing takes time, writing new code takes time, implementing and testing the changes on the PTR takes time. There's no MMO magic wand that devs can wave and just make changes happen. Patience is a virtue.
----
			
{| class="blockquote"
Learn how to code and you will understand why it takes so long. Making one change affects multiple systems that then all need changing. Finding the bits of code that need changing takes time, writing new code takes time, implementing and testing the changes on the PTR takes time. There's no MMO magic wand that devs can wave and just make changes happen. Patience is a virtue.
----
			
Seriously, I can't believe people are acting like these requests are unreasonable. Look at the issue of Archivos shoulder specials:
 
https://forums.wildstar-online.com/forums/index.php?/topic/118377-archivosha-55-vitaphagic-shoulderplate/page-2
 
It's not some nebulous or hard to fix bug like optimization/fps/lag. It's literally changing an incorrectly assigned special (healing proc for tanking gear). In fact, it's not even something that fell through the cracks in triage. According to post #38, it's been fixed for over a month. The holdup is that nobody knows when the fix is supposed to be pushed out nor can they decide whether we should have the fix at all until *all other itemization fixes* are done.
 
This is what OP and people like me want to see changed. If you can't fix a bug or don't have the time to fix it, fine. I don't like it but I recognize the resource constraints this company is under. If a change is large enough to affect a bunch of things in game, then sure, be more cautious in releasing it. 
 
But small bug fixes with negligible impact on the rest of the game like the example above are consistently being lumped in with larger drops and changes, delaying quality of life improvements for no reason other than someone likes to group things up and release them in large chunks rather than hotfixing things.
 
So this is a plea to Carbine: please put more bug fixes into patches or weekly updates instead of withholding them from us. There are plenty of them that could qualify. Don't be so trigger happy to put a fix in a drop, then forget about it for 3 months. 	
----
			
{| class="blockquote"
Seriously, I can't believe people are acting like these requests are unreasonable. Look at the issue of Archivos shoulder specials:
 
https://forums.wildstar-online.com/forums/index.php?/topic/118377-archivosha-55-vitaphagic-shoulderplate/page-2
 
It's not some nebulous or hard to fix bug like optimization/fps/lag. It's literally changing an incorrectly assigned special (healing proc for tanking gear). In fact, it's not even something that fell through the cracks in triage. According to post #38, it's been fixed for over a month. The holdup is that nobody knows when the fix is supposed to be pushed out nor can they decide whether we should have the fix at all until *all other itemization fixes* are done.
 
This is what OP and people like me want to see changed. If you can't fix a bug or don't have the time to fix it, fine. I don't like it but I recognize the resource constraints this company is under. If a change is large enough to affect a bunch of things in game, then sure, be more cautious in releasing it. 
 
But small bug fixes with negligible impact on the rest of the game like the example above are consistently being lumped in with larger drops and changes, delaying quality of life improvements for no reason other than someone likes to group things up and release them in large chunks rather than hotfixing things.
 
So this is a plea to Carbine: please put more bug fixes into patches or weekly updates instead of withholding them from us. There are plenty of them that could qualify. Don't be so trigger happy to put a fix in a drop, then forget about it for 3 months. 
----
			
{| class="blockquote"
Learn how to code and you will understand why it takes so long. Making one change affects multiple systems that then all need changing. Finding the bits of code that need changing takes time, writing new code takes time, implementing and testing the changes on the PTR takes time. There's no MMO magic wand that devs can wave and just make changes happen. Patience is a virtue.
----
			
{| class="blockquote"
Yeah 4 months for fixing MAJOR issues is pushing it mate.....
----
			
{| class="blockquote"
Let's completely forget that they have been fixing MAJOR issues, mate... some fixes may not be coming as quickly as you, personally, would hope, but Carbine has to prioritize. Agreed, some of their priorities have been a bit bewildering, but the point is fixes are in the pipeline; it can't all be done at once.
 
 
Other sprinters are better and faster than I am, does that mean I should automatically run as quickly as they do every time I run? No. I can only be expected to do as well as I am physically capable of doing, with incremental improvements. I didn't "flame" anybody, I simply stated a fact. The hypocrisy with your statement is astounding.
----
			
{| class="blockquote"
Let's completely forget that they have been fixing MAJOR issues, mate... some fixes may not be coming as quickly as you, personally, would hope, but Carbine has to prioritize. Agreed, some of their priorities have been a bit bewildering, but the point is fixes are in the pipeline; it can't all be done at once.
 
 
Other sprinters are better and faster than I am, does that mean I should automatically run as quickly as they do every time I run? No. I can only be expected to do as well as I am physically capable of doing, with incremental improvements. I didn't "flame" anybody, I simply stated a fact. The hypocrisy with your statement is astounding.
----
			
If having fixes ready for a month and not pushing them out to your customers makes ANY sense, I'm sorry, please explain it to me so I can comprehend it.
 
As soon as a fix is done and has been tested it should be pushed, not delayed for some bi weekly patch day (whatever this means these days because they never were bi weekly)	
----
			
{| class="blockquote"
If having fixes ready for a month and not pushing them out to your customers makes ANY sense, I'm sorry, please explain it to me so I can comprehend it.
 
As soon as a fix is done and has been tested it should be pushed, not delayed for some bi weekly patch day (whatever this means these days because they never were bi weekly)
----
			
{| class="blockquote"
ICC Comm Issue - https://forums.wildstar-online.com/forums/index.php?/topic/118708-iccomm-update/
 
Support only Archivos weapons - https://forums.wildstar-online.com/forums/index.php?/topic/119127-archivos-cw-5-grs-phageblade/page-2
----
			
{| class="blockquote"
 
There is a patch scheduled for tonight (pending the usual caveats that the build has to not explode, etc). In it:
ICComm - some fixes for this
Archivos weapons should now be properly affected by the primary stat chip types
 
----
			
{| class="blockquote"
Let's completely forget that they have been fixing MAJOR issues, mate... some fixes may not be coming as quickly as you, personally, would hope, but Carbine has to prioritize. Agreed, some of their priorities have been a bit bewildering, but the point is fixes are in the pipeline; it can't all be done at once.
 
 
Other sprinters are better and faster than I am, does that mean I should automatically run as quickly as they do every time I run? No. I can only be expected to do as well as I am physically capable of doing, with incremental improvements. I didn't "flame" anybody, I simply stated a fact. The hypocrisy with your statement is astounding.
----
			
{| class="blockquote"
This was Carbine's initial strategy.  Just get the bug fixes out as fast as possible literally weekly at times.  The reality seems to be that this created a cascading effect where a bug fix would be applied and cause more bugs when another was applied.  That led to them dropping the monthly patch cycle they were initially trying for and moving to quarterly.  The only bug fixes they can push out are ones that have literally been tested not just in the current game build, but against everything coming out afterwards.  That takes time, and means that you have to literally, from back to front, test all your fixes against every other fix, addition, or correction you put out.  That's the internal build that they bring up. 
----
			
{| class="blockquote"
What your talking about was for their Content Drops.  They made the statement of doing a content drop every month with new things.  The issue was that QA was overwhelmed and letting to many bugs through to meet a timeline, so they moved to a Quarterly drop cycle to allow QA more time to fix bugs with the new content.  The problem here is that as seen with Drop 3, while not as bad as Drops 1 and 2, still had a sizable number of bugs that needed fixing before going to the playerbase. 
 
The only comment Mike Donatelli made on Bug/hotfixing was on the on the Nexus Report in September that these should be coming out quicker, but they clearly aren't at this point. Things seem to keep getting directed to drop conent patches which they shouldn't be.
----
			
You highlight 2 issues going into the "Biweekly Patch" one of which was reported "fixed" by the devs since November.
 
I'm not saying what they are doing is wrong or not warranted.  I'm not saying that they don't know how to program.  I'm not saying that they don't know how to patch.  I'm saying their current process needs improvement and is causing issues to go on longer than they probably should to the public eye.  Whether that means they need to change the way the identify and prioritize issues, or if its a simple scheduling change getting new iterations out quicker.  Without knowning fully what their current process is, I can't really comment on it except to say it needs work, and neither can you defend it saying its perfect, because you don't know.
 
You speculate that this is how they manage and fix changes, but you could be completely off as well, since none of us really know the answer,  Your speculating just as I am.  Except im not speculating on what they are doing and defending, im saying whatever it is they are doing is taking certain things to long to get to market.  I would love to have a healthy discussion on how to improve that process, however it needs the developers to explain and be involved in the discussion.  And I do want to have that discussion about why these things are taking so long, and unless a carbine employee steps in and says why, we can only ask that it gets improved and things get pushed quicker than a 2 month time frame.	
----
			
{| class="blockquote"
You highlight 2 issues going into the "Biweekly Patch" one of which was reported "fixed" by the devs since November.
 
I'm not saying what they are doing is wrong or not warranted.  I'm not saying that they don't know how to program.  I'm not saying that they don't know how to patch.  I'm saying their current process needs improvement and is causing issues to go on longer than they probably should to the public eye.  Whether that means they need to change the way the identify and prioritize issues, or if its a simple scheduling change getting new iterations out quicker.  Without knowning fully what their current process is, I can't really comment on it except to say it needs work, and neither can you defend it saying its perfect, because you don't know.
 
You speculate that this is how they manage and fix changes, but you could be completely off as well, since none of us really know the answer,  Your speculating just as I am.  Except im not speculating on what they are doing and defending, im saying whatever it is they are doing is taking certain things to long to get to market.  I would love to have a healthy discussion on how to improve that process, however it needs the developers to explain and be involved in the discussion.  And I do want to have that discussion about why these things are taking so long, and unless a carbine employee steps in and says why, we can only ask that it gets improved and things get pushed quicker than a 2 month time frame.
----
			
I wasn't going to comment on this, but it seems to be a hot issue that keeps recurring, so...
 
Let me share some of our process that makes it take so long to see fixes.
 
First off, this game is big, so sometimes we miss things, we don’t see the bug until after it has gone live.  We can’t possibly play every race/class combo to max level and do every quest and use every skill on every patch.  It’s simply not possible.  We test the things that we think are most likely to show us bugs in the most number of systems.  Sometimes that means that things that are blatantly obvious to someone “actually playing” the game gets missed.  It’s an unfortunate reality.
 
First, we see a report on the forums about something wrong, or we experience it first hand, or some other way that the bug gets reported.  There are many things reported, so we try and prioritize what is impacting the most people, and how much it is impacting them.  This is purely a judgment call, and someone else would almost certainly make a different judgment, especially if the particular issue is affecting you. :)
 
Next, our QA department tries to reproduce the bug.  You would be absolutely amazed at the number of bugs that only show up in a live environment, that cannot be reproduced in a debug environment.  Sometimes these bugs are timing related, sometimes they are related to the number of players online, sometimes they are related to lag, some of them are related to the servers having been online for 7 days straight, and other times, we simply can’t explain why….
 
Once we know how to reproduce the bug, It’s evaluated for risk level in fixing the bug.  If it’s low risk, we’ll assign it to be fixed in the next patch.  If it’s medium risk, it’ll be assigned to the next drop, and if it’s high risk, it will be assigned to the drop after next so that it can have extensive testing.  Once we’ve decided on its risk level, it will be assigned to someone to fix.  We’re quite busy already, and so the bug will get fixed according to its priority relative to the other tasks that person is working on.  Also, no bug takes 10 minutes to fix, I’ve had bugs that took us 3 weeks to even find the bug, and then a few more weeks to actually write the code to fix it.
 
Once the bug is fixed, we wait for a full build to have QA test it.  Full builds happen nightly.  So this will be at least the next day before it’s tested.
 
Once it’s tested and verified fixed (if it’s not fixed, rinse, and repeat, as many times as necessary), we evaluate the risk again, and make a decision as to whether it gets deployed to the next drop, or the next patch, or wait for a full drop cycle.  This is tricky, everything is interconnected, and we don’t want to break the live game worse than if we didn’t fix the bug.  Assuming we decide this needs to go to live as soon as possible, then it’s marked for promotion to the live environment.  And it will be put in the queue.
 
Once it’s promoted to the live environment, we have to wait for a full build again, another overnight process.
 
Once that live candidate is built, then QA tests it again now in the live mirror environment.  And remember, they’re not just testing what was fixed, they’re trying to test and see if the fix affected anything else.
 
Assuming that checks out, then that build will get scheduled to be deployed.  We don’t just want to deploy at any arbitrary time, players would hate having us shut down the servers every day for a patch, so we schedule them during the off-peak times, and we try to only shut down the game once a week or less, ergo our every other week schedule.
 
Now given that process above, I listed 2 overnight processes, which means a bare minimum of 3 days for a bug.  If we were only fixing one bug at a time, we could fix it weekly.  Looking at our last set of patch notes (https://forums.wildstar-online.com/forums/index.php?/topic/119985-bi-weekly-update-12162014/), there were over 150 issues fixed.  That takes a lot longer to process than 3 days, and yet that patch was only 7 days after the previous patch.  It also clearly didn’t take 3 days per issue, because that would have been over a year (450+ days), many issues are being worked on in parallel, but not all of them can be.  And most things take significantly longer than 3 days as well, with people working on multiple bugs, and there being a backlog of things to work on, it can take weeks before we even look at them.
 
Now some of you will remember, that we have actually done hotfixes within hours of finding game breaking bugs.  In order to do that we forgo several important things.  First off, if it’s game breaking, reproducing it is usually not an issue.  We know exactly what the issue is.  Next, we fix the bug with very little regard to what other systems it could impact, we try to pay attention, but when it’s that important some precautions go to the wind.  If it’s something that can just be disabled until a “proper” fix can be done without completely breaking the game, that’s often what we’ll do.  We don’t do a full build, we do what’s called an incremental build, which only rebuilds the parts that changed.  Incremental builds sometimes don’t work “quite right”, they are usually just fine, but every now and then you get a weird bug out of them that breaks in a way that isn’t in your code.  So since it’s a bug that’s not in your code, you can’t actually figure out what’s wrong, doing a full build will make these weird bugs go away.  That’s why we don’t use them for deployment builds normally.  Next we do the bare minimum of testing, is the “game breaking bug” fixed, we don’t really check other systems for other issues, we do the minimum of testing that we think is reasonable.  And then we cross our fingers and deploy it.  We hate doing this, as often as not it comes back to bite us.  And even with throwing all that caution to the wind, it’s still usually several hours to get a new hotfix out, and it almost certainly means we introduced new bugs, so we’ll be doing it again in the days that follow.
 
I hope this helps to shed some light on why things that seem so simple, aren’t.  And why things can take longer than one would expect to fix.  This game is extremely complex, with literally millions of lines of code and that’s only a fraction of the game, there is also art, and sound, and text and …  The list goes on and on…
 
Huh... I lost my carbine tag again...  I'll get that fixed...
 
But for those who don't know, I'm the Lead Server Programmer.	
----
			
{| class="blockquote"
SNIP
----
			
Actually I did have a dog that ate my homework years ago, ripped to bits maybe a bit more accurate!
 
All MMO's have bugs, some still have the same bug after 10 years, some you can forget about and some you can't! Pick the top 20 bugs those that play the game moan about, those that effect the day to day enjoyment of Wildstar and fix first.
Give the players more information and stop trying to whitewash over it all....and please please stop trying to put out the fire with gasoline.   	
----
			
{| class="blockquote"
Yes, we all know that bugs are not fixed over night. Everyone needs to stop acting like no one in the gaming community knows this. Its insulting.
 
[snip]
----
			
{| class="blockquote"
I'm sorry you feel that way about my response.  I was trying to give some insight into the process of how we go from bug to patch.  I wasn't making excuses, and I don't own a dog. :)  I was simply trying to explain the process.
----