119354-t4-hour-hotfix-that-didnt-happen

== Content ==
			So i just waited in front of my computer . waiting to log in and at every hour i try to log in only to find that the maintenance has been pushed by an hour . This normally would be fine but being pushed back 30 mins and then one hour 3 times is a bit excessive . I just sat in front of my computer doing nothing waiting to log on . Do people like me a favour and just say youre doing maintenance for 6 hours and be done with it . saves me alot of time. and the best thing about this is the hotfix did not even go through and you gusy are reverting it back to the previous build which means there's gonna be a nother '1.5 hour hotfix' thhat turns into a 5 hour nothing . I love this game but these things that you guys are doing is making it hard to come back.	
----
			Carbine: "Applying Hotfix Servers will brb !" Avatus: "Im thinking not so much!" 	
----
			
Hello,
 
Our apologies for the extended downtime. Unfortunately some times things happen that do prevent us from bringing the servers back up at the expected time, things that we can't foresee until they're in a live environment (as Virtue posted here). We try our best to keep the information to players as updated as we can. 
 
We thank you all for your patience. I understand it's frustrating (it frustrates us too!) 	
----
			thats all fine and dandy . just stating that the maintenance would take 6 hours from the start woulda been good . loving this game is bad then apparently . will temper my expectations for carbine actually keeping to what they say then 	
----
			
Carbine is not responsible you not having anything to do.
You, however are responsible for filling your own time.
 
Carbine is doing what they're supposed to do:
 
Maintenance
Find and solve problems
announce issues they encounter	
----
			
Side note, things should be up now, if you wanted to log in. =) 
 
- Team WildStar 	
----
			
{| class="blockquote"
thats all fine and dandy . just stating that the maintenance would take 6 hours from the start woulda been good . loving this game is bad then apparently . will temper my expectations for carbine actually keeping to what they say then 
----
			http://youtu.be/zjxLml-4_MM	
----
			
If we get boomboxes again out of this, could you please make the rng bit moar random. I'd prefer different useless items instead of copies of the same useless item.
 
Edit: Did a quick mining round and found total 2 full nodes, no unbugged deposits at all. (But saw someone else mining, perhaps I was just too late.)	
----
			
{| class="blockquote"
thats all fine and dandy . just stating that the maintenance would take 6 hours from the start woulda been good . loving this game is bad then apparently . will temper my expectations for carbine actually keeping to what they say then 
----
			Most times, Carbine's been very good at staying within the amount of time they said would take for maintenance.  Today just happened to be one of those days that things didn't go right.  One thing I learned from playing other MMO's is that *cupcake* like this happens, and it's not necessarily the fault of the game company, it just happens.  Wildstar's been way more consistent with their downtime estimates than others I've played.  Experience from other MMO's has taught me to not count on Wildstar's estimates(even though they usually stick to them) that way I'm pleasantly surprised when the game's up when they say it will be, and only thinking "business as usual" when they aren't :D	
----
			
{| class="blockquote"
Carbine is not responsible you not having anything to do.
You, however are responsible for filling your own time.
----
			
{| class="blockquote"
Crap response.  The person is well within their right to be a bit peeved about this.  Being kept on a shoe string means you can't just arrange something that you could have done if you'd known you had 4-6 hours and not 30 mins to 1 hour.
----
			
Elaborating my little parable above:
 
"Sorry, I'm going to be late for dinner. Looks like a traffic jam ahead - must be some new road construction, so I'm going to be an hour late."
 
An hour later:
 
"Sorry, still stuck in traffic. Must have been an accident ahead, but I'm sure they will have it cleared soon. But I'm going to be yet another hour late."
 
Said hour later:
 
"Sorry, it looks like the road was flooded. I'll have to make a detour to get home..."	
----
			
It doesn't excuse the response though does it.  We all know there are unexpected problems that occur but the response I highlighted is completely useless and basically done to further frustrate the OP.
 
The OP is clearly frustrated by what has happened and may not have experienced a downtime before in an MMO so didn't know whether the "30 minutes" quoted was going to suffice which clearly it didn't.
 
In future though they may well just go and do whatever for a few hours then come back but after thinking they'd have their spare time to play and not being able to it's understandably frustrating.
 
As for your example..this is software, not a traffic jam.  You can rollback at any point.  It's a decision to be taken...stuck in a traffic jam you can't move.  If you plan for a 30 minute fix and it starts taking 1hour 30 and you hit more issues, pull the plug and rollback and fix your issues in the PTR/DEV again.	
----
			
Hi folks, 
 
There's no need to argue over responses given within this thread from other players. We did apologize numerous times for the downtime, and we understand it was frustrating for players - and for us too. We do try to keep on target with our projected down times, but as mentioned above every so often they have to be extended and we do our best to let players know this. 
 
- Team WildStar 	
----
			
Yes but maybe someone at Carbine needs to take a look and start learning what rollbacks are and how quickly to do them.  The exploit from Drop 3 that didn't get a rollback is another example.  If something is causing problems and taking too long to put in, IT'S PROBABLY WRONG!!!!
 
Don't just stick it in and hope it works, no wonder there's bugs appearing that don't exist on PTR if they're trying all sorts of stuff on the fly to get a patch in.
 
It would do someone there some good to take a look at these things and say "you know what, it's not working..roll it back, get the servers back up and we need to go back to DEV and try again".
 
Stick an hour downtime in, that's your drop dead time, if it's not in roll it back.  Work on what's broken, plan another 1 hour down time or maybe two because we're really clever and it's not as easy as first thought, then you've given contingency for the problems coming back up too and your customers can prepare for that downtime as Olivar said..then it's the customers fault if they know of the downtime.	
----
			
{| class="blockquote"
Yes but maybe someone at Carbine needs to take a look and start learning what rollbacks are and how quickly to do them.  The exploit from Drop 3 that didn't get a rollback is another example.  If something is causing problems and taking too long to put in, IT'S PROBABLY WRONG!!!!
 
Don't just stick it in and hope it works, no wonder there's bugs appearing that don't exist on PTR if they're trying all sorts of stuff on the fly to get a patch in.
 
It would do someone there some good to take a look at these things and say "you know what, it's not working..roll it back, get the servers back up and we need to go back to DEV and try again".
 
Stick an hour downtime in, that's your drop dead time, if it's not in roll it back.  Work on what's broken, plan another 1 hour down time or maybe two because we're really clever and it's not as easy as first thought, then you've given contingency for the problems coming back up too and your customers can prepare for that downtime as Olivar said..then it's the customers fault if they know of the downtime.
----
			
Problems are bound to arise. Sometimes those issues will inconvenience us. What you should be looking for is how a company deals with problems. If they have done a good job to mitigate issues we don't see anything more then minor inconvenience.
 
In this case I feel they have done everything they can. I'm glad they kept issues to a minimum. :)	
----
			Ok, so I'm a little confused. There was a patch, then it was pulled? So no actual changes has happened? What's with the bit about resurrection? If we didn't patch/log in between deployment/reversion, is that going to matter to us?	
----
			
{| class="blockquote"
You have no clue about server deployments on this scale...
It's not that easy as "just roll it back"
----
			
{| class="blockquote"
Working for a global company, I think you'll find I do.  There should be rollback procedures in place, not just a mess of hands going in changing things to plug the gaps as it goes in or to get it back out.
 
You don't work at Carbine to my knowledge so you can't comment on how easy or hard it is to do, fact is the rollback should always be prepared for every release.
----
			
{| class="blockquote"
Ok, so I'm a little confused. There was a patch, then it was pulled? So no actual changes has happened? What's with the bit about resurrection? If we didn't patch/log in between deployment/reversion, is that going to matter to us?
----
			I think I'd rather wait a couple of unexpected hours than to fear losing progress every time there's a hotfix, patch, maintenance, etc...	
----
			So you don't have any rollback procedures should they be needed in place before you go live with things?  If there is then how hard is it to state a guaranteed downtime window and initiate the rollback if it's not successful, I'm not talking about rocket science of deploying to multiple clusters and servers, blades etc etc here, I'm talking really basic things.	
----
			
{| class="blockquote"
So you don't have any rollback procedures should they be needed in place before you go live with things?  If there is then how hard is it to state a guaranteed downtime window and initiate the rollback if it's not successful, I'm not talking about rocket science of deploying to multiple clusters and servers, blades etc etc here, I'm talking really basic things.
----
			
{| class="blockquote"
newsflash : Game servers and software are not "basic things".
You cannot just ignore the fact you're dealing with a cluster or shard-setup.
These things are delicated, and simply initiating your deploy in the wrong order can destroy your environment.
 
So again, this is not as easy as you think it is.
----
			
{| class="blockquote"
Yes, that's why you have to have the rollback tested and in place and in the correct order.  Just like other complex systems, you do things in the wrong order and it will break stuff and not work.  That's why the rollback should be tested beforehand.
 
I've not once claimed it's easy, it's complex but it should be in place if it's required.
----
			Yes things don't always go swimmingly...rolling back is the same as releasing code (unless you're just bunging a backup back in).  At the same time you can't have a 50% strike ratio either and expect people to not post things like this thread.	
----
			
{| class="blockquote"
Yes things don't always go swimmingly...rolling back is the same as releasing code (unless you're just bunging a backup back in).  At the same time you can't have a 50% strike ratio either.
----
			
{| class="blockquote"
Yes, that's why you have to have the rollback tested and in place and in the correct order.  Just like other complex systems, you do things in the wrong order and it will break stuff and not work.  That's why the rollback should be tested beforehand.
 
I've not once claimed it's easy, it's complex but it should be in place if it's required.  The people are good enough to write a game, they're good enough to write a rollback.
----
			
{| class="blockquote"
How do you know they don't have a rollback in place for when/if it's required?  A hotfix "OOPS!", IMNSHO, doesn't seem like a thing that needs the rollback used for.  Hell, if they didn't use rollbacks for the exploits, why the heck should they use them for a hotfix "OOPS!" :P  So we lost a couple of hours of gameplay this morning, it's not like that isn't unexpected in MMO's.  And was quite short compared to other times/games this happens with.  
----
			
{| class="blockquote"
Dude...rolling back is not even remotely close to "releasing code".
Whatever your patch done, is not even guaranteed to be reverseable.
----
			Moral of the story.... Charge as no idea what he is talking about, and Chua is wasting his time arguing.	
----
			
{| class="blockquote"
Like cutting a cable in half when you don't have another in the cupboard?
 
Enlighten me, what can you do that you can't put back.
----
			
{| class="blockquote"
do you honestly think deploying a new server patch is in the lines of "git pull"?
You're talking configuration changes, database updates, Windows update patches (goodluck rolling that one back without trouble), external dependencies that maybe got updated and broke something for you.
Let's take a "global company" one: Upgrading from .NET 3.5 to .NET 4
----
			
{| class="blockquote"
Moral of the story.... Charge as no idea what he is talking about, and Chua is wasting his time arguing.
----
			
{| class="blockquote"
Why can't you do this?
 
I fully understand what you're saying that there's tons of variables etc and it's a royal pain in the backside but all of the external libraries, server patches etc etc can and should be tested.
----
			
{| class="blockquote"
If I have to explain this to you, I highly doubt you work at a "global company" or even have the slightest clue about server software architecture.
----
			
{| class="blockquote"
newsflash : Game servers and software are not "basic things".
You cannot just ignore the fact you're dealing with a cluster or shard-setup.
These things are delicated, and simply initiating your deploy in the wrong order can destroy your environment.
 
So again, this is not as easy as you think it is.
----
			
{| class="blockquote"
That's my point, you don't need to explain it, I've done it.
----
			
{| class="blockquote"
I didn't say they don't have, I hope they do.
 
What I said was they should maybe be a bit more stubborn with when they initiate them.  Purely in the context of what the OP is talking about and stringing people along if the downtime is set to last 1 hour then you know your rollback is 30 mins, after 30 mins if it's not working take it back out.  Alternatively, give a message out saying apologies there's an issue, we're extending for 1 hour and if you haven't solved it in 1 hour then roll it back.  Don't do the same over and over extending and extending.
----
			Yea because websites don't use binaries.	
----
			
{| class="blockquote"
Yea because websites don't use binaries.
----
			
{| class="blockquote"
I don't get why people are asking about a "rollback" because that's pretty much exactly what they did.  In a rollback situation you either replace the build (sometimes manually from the command line if the server is a total goner, had some fun sleepless nights with that) or go full wipe and restore a backup.  
----
			
{| class="blockquote"
If you're talking ASP.net I can let is slide cause that's indeed a bit more then updating pages.
If you're talking SAP, please start running.
----
			
{| class="blockquote"
Hello,
 
Our apologies for the extended downtime. Unfortunately some times things happen that do prevent us from bringing the servers back up at the expected time, things that we can't foresee until they're in a live environment (as Virtue posted here). We try our best to keep the information to players as updated as we can. 
 
We thank you all for your patience. I understand it's frustrating (it frustrates us too!) 
----