several technologies in the past have preceded their science that explains them for example the steam engine was before the thermodynamics so it's doing better theory for ganz is something that could still be innovated on in the future so here is a taxonomy of generative models from in Goodfellas new ribs tutorial apart from Markov chain Boltzmann machines and Markov change and are the stochastic networks we have pretty much covered everything else we've covered Nate may fix Lauren and how do you exchange of variables scale the flow


models or really be models all these are explicit density models and then we also covered approximate in steam models vary from our encoders the variation lower bound and then recovered implicit density model estate they can other models that I'm not being covered are not particularly popular or very used so that's the reason we focus on the more popular ones and if you have if you're if you have been and trained density models and you're figuring out which density model you should be using here are some pointers if you only care

L8 Round-up of Strengths and Weaknesses of Unsupervised Learning Methods -- UC Berkeley SP20


about the density estimates disco for our aggressive models you don't worry about sampling time here if you care a lot about sampling times in autoregressive may still be fine if your sequences are not that big or if you use lightweight models but if you really cannot afford to wait for the sampling time you really want really fast samples but you still want to go for a density modeling you could think about using Vikan regressive models like paralytic so CNN and you could also think of doing latent space modeling like like latent


space or like a week you BAE you may probably not even needed quantization bottleneck it could still work with like continuous values and so models are also pretty billing for modeling continuous value data that density estimates for continuous value data especially even when they're actually continuous and it's hard to figure out how to even quantize them so so that that's that's another interesting aspect of flow models and if you also want to think about how how to have like representations and also sampling but


you want to have a simple possible model v's with factorize decoders maybe the natural drugs so given given these appealing properties or density models like when would you use cans you would use guns when you really care about having good samples and you have really really large images high-quality images for and you don't want something photorealistic you have a lot of conditioning information like pose or the class or edge edge maps and you just want to add texture to them cans are really good in these initial image


translation problems or rear the video and if all they care about is perceptual quality and controllable generation and you don't have a lot of compute this is often the case for any any kind of start up again it's like the best choice to go for so that's it for generative models next let's look at South provides representation learning it which is our final topic so south supervised image classification has seen rapid advance in the last one and a half years just the end of 2018 the top one accuracy of image net linear classification


benchmark was 48 percent and now it's seventy six point five percent so this rapid advance has been made in multiple labs because of this mode of learning called contrast to learning and contrast the learning task can be simply summarize this a dictionary lookup task and there are two ways to do this pretext contrasted learning which is you either build it as a predictive coding task or you build it as an instance discrimination task and in predictive coding you have multiple mechanisms to do that once you either used end-to-end


mechanism or you use the momentum decoder momentum encoder make using the momentum encoder for the keys and the predictive coding success story has been achieved in the contrast operator coding or CPC particularly the CPC version two and and and the instances combination success has been achieved in moko and sim clear moko means momentum contrast and Sinclair's into an instance contrast they use the corresponding mechanisms of contrast learning so let's look at CPC version two moko and Sinclair in terms of their positives and the negatives so


CPC version two we're doing spatial contrast prediction so that principle is very generic and it can apply to any morality or domain so you don't need to know the underlying data augmentation in variances in this work and it can be considered as latent space channel tomorrow and also it's much easier to adapt for audio video text and perform multimodal training disadvantages it splits your input into a lot of patches or frames or even audio chunks and therefore your inputs are now your inputs are now basically split into


a lot of different parts that you have to carefully delineate and you also need to carefully pick what part are you predicting from what so that involves a lot of design choices to make type of parameters that you can only know by trial and error so that makes it really hard for you to use it on a domain or task that you don't really understand well and then you require multiple forward passes for these smaller versions of the inputs now and so that means that you be pre-training on something much smaller but potentially


fine-tuning are much larger versions of the sequences or images so this may not be an optimal thing to do when you're doing local predictions local spatial predictions Bosch num is hard to use so applying mass ROM is hard but then you really want to use batch room for a downstream task so that makes CPC version too little sore in sense it's not particularly suitable for downstream tasks if you really care about state-of-the-art performance and finally the splitting process mechanism is very slow on a on a matrix multiplication


specialized hardware like GPUs so it's because you do a lot of reshapes and transposes and so it's never an optimal thing to do so here's the summary of moco one of the main advantages of moco is it is very minimal so it's very easy to use and replicate and it has no architectural change can be easily applied for downstream tasks there is no notion of a patch and it's distilling in variances for images using data augmentations and so the pre-training procedure looks very much like supervised learning and therefore it can


get comparable or even better results and the momentum encoder memory bank can assume adds a lot of stability to the training and decouples back size from the number of negatives and therefore this lets you train with way fewer GPUs than what's needed for CPC or like methods the disadvantage with moco is that because you introduce momentum and date you need to figure out what's the right decay rate for that and that has an extra type of parameter and another disadvantage is in image augmentation the invariances may not be applicable to


other modalities so this may be in method this works only for a visual image recognition and finally let's look at simply er which can be considered as an end-to-end version of Tomoko where you just look you're using all the negatives from your batch and there is no momentum encoder so advantages or sim clear are the same as that of moko with the additional advantage that you don't have a momentum in kora now so it's going to be asked minimally supervised learning but the disadvantage is now you just need really large batch sizes


because you need a lot of negatives because moko decouples the negatives from a bad size it doesn't need as much compute as sim cleared us and similar to moko they documentation invariance may be very specific to image recognition so in terms of future work left for sauce provision the gap between some supervised learning and supervised learning is to not close if you consider just the same amount of compute training time and the same candidate augmentations use so and also fine-tuning to downstream tasks the


gains are not significantly high enough that the paradigm shift has been made in vision so that way maybe new objectives are also needed and finally all these sub supervised successes have relied on using image net and it's not clear if supervised learning we just work from images in the wild or from the internet which is really the dream and which is really why people wanna do something so that's it for like subspace learning as in in terms of utility for downstream tasks let's look at always learning in the context of


intelligence like being able to act in an environment so here is a video of this quake3 game where yeah like that you can see some characters and then you can see some bullets that there are going to be fired and you know you see all these different walls and fires and other characters and when you're looking at all this you're able to already accurately parse the scene make sense of what's going on and you're also able to clearly separate out the objects from what's not objects and and so we need to be able to do that as well we shouldn't


be working at the level of pixels we should be able to predict the future in a much more semantically in space and so modeling the pixel space for these high dimensional videos is really hard and in order to build really dungeon agents which that can planning faster than real time we should be able to do it in the lane space that's more abstract so how do we do that what is the right kind of abstraction to build and how do we learn role models in that Lane space that can this ignore noise and work in a much more semantic space it's really the


hardest question to think about and this has also been summarized multiple times by omnicom that if you have very good internal world model you'll be able to plan with it and a wide lot of mistakes there and our relation usually makes and and how to do that is one of the most important questions so if you want to have the overall view of subspace learning across all these different problems for image recognition we saw or assesses like city scene workers in clear moco version to transfer learning it works really well in language but the


exact details will be covered in a future lecture and transfer learning and vision also works reasonably well now been shown in CPC and moco but there's like close to nothing in terms of how to use of supervised learning for RL so that's the very ripe area for future and then as far as like you know using sound supervision in the context of general intelligence is considered its it's potentially going to be extremely useful in the context of transfer learning and learning use of abstractions for planning or imaginations so that's just a lot of


work to be done there so that's that's that's it for the summary of the class it's pretty much ends with our original motivation which is how do we build this intelligence cake and and a lot of it is gonna be done through supervised learning and and so in terms of future lectures they're gonna look at more applied topics which are not falling into the main main main lecture stream which is that we be looking at semi spread learning we'll also be looking at the whole area of one square learning for language which is language models


and bird and then finally we look at how representation learning or supervised learning has been applied in the context of reinforcement learning so and and we will also cover things like how to do unsupervised distribution alignment that is given completely to different data sets with a lot of common information how do we align the two manifolds together and without any prior data and you see how generative models and unsupervised learning can be used in the context of building compression algorithms so that's the next next


[Music] okay so hi I'm Lucas easy one first of all thanks to debug and to be a developers for inviting us today so what we will try to do today is share some of our experiences when it comes to IP architectures we'll also try to give some recommendations about what to do and what not to do when you're designing an architecture in a larger enterprise environment we will try to give examples where we can from our own experience and you will see we will cover actually several topics so feel free to ask any questions if you know we go cross one

Building high performance and scalable architectures for enterprises—Luka Samaržija & Ivan Sokol


topic too fast because number of topics that we want to share so okay let's start so actually what is an enterprise IT landscape and how does it look like you probably guess that it's big given and I we work in telcos most of our careers which is a really big act landscape but just to make it more clear I'd like to compare it with the buildings and let's say a neighborhood so it looks like something like this you know it's a being it's modern it's neat it's beautiful I'm kidding of course it doesn't look like that at all probably


only chief architects thing that looks like this actually it looks something more like this or this or this if you're lucky the reason is that most enterprises have been building their IT for the last twenty thirty years and doing it mostly by adding new systems on top of existing ones rarely decommissioning your ones so now you have a really big combination of really cool and modern stuff and some really uncooled and all that stuff that's it that actually makes it really interesting but also challenging so but


what happens at one point since you're building building building on the other it starts to crumble and you actually have two choices you need to you need to change something or as you like to call it these days do a transformation project and then you can do it in two ways basically you can go all the Intuit Big Bang changed 80% of your landscape or you can do it progressively in our experience I have done one transformation that was a big bang approach let's actually first don't don't do Big Bang transformation


projects try to do it one step at a time and progressively if possible so how can you do it well first you need to identify what are your biggest pain points what is the what is the thing that is most troubling your business with the current IT landscape then think of a better way to doing that when you do that take a step back because you're probably already going to white and you're looking at the project of three to four years you can't do all of it do a minimum of what you can think think of but but it makes sense that it really


solves the biggest issue that you can and that you can integrated with the existing systems that you have so in order for you to do that there are actually three things that you need to consider and have in mind for us in our experience microservices architecture is a must in doing that why not just because it's a hype and everyone in doing that but because you can also release in a lot of books that talk about macro services architecture it's as people say so are done right it's finally finally we have the technology and the means to do


service-oriented architecture in a good way to make it scalable and maintainable so that's why macro services architecture is a must but when you can really do a lot of wood or doing a lot of bed is a in integration part so you need to be smart when it comes to integration patterns and also is in all development unit who developed development practices system otherwise won't be able to tactical so those are actually free areas that we will try to cover in our presentation first the micro services architecture the main


message that I want to convey here is the micro service isn't the support application now don't get me wrong I have nothing against people would I like it actually but though many times I've seen someone will the Java spring boot application deploy it on the docker container and says I have a macro service you don't have a micro service unless that micro service has data when we come to the definition of a micro service in its essence it's a small autonomous application that can handle one domain independently autonomous and


independently you can't do that if you don't share data you can't do that if you're creating a facade on my or proxy my cursor that keeps baiting legacy and this query in that legacy for all day I mean you can do it and that there are uses for such such applications but you can forget about any kind of performance or scalability if you query an existing legacy banking systems for every time someone the queries your API on a micro service and needs to get some data that's the most important peak we will talk later


even will talk about what are good ways to get data out of legacy in inter macro circuit so I'm going to that you'll hear that later other thing that is important is how you design your micro service what is important is a lot of times people have a set of functionalities you put it in a micro service and you you just go from there it's a great way to build a my product what that means is build a micro service that's actually monolith just your deploy it on docker and the quality Mac series in micro service design you should really look at


domain driven design that's another concept you have great books about it especially the part you have a book by Eric Evans domain driven design when he talks to bounded context but what that means actually in a nutshell is that when you design a data model for an enterprise you can't design one data model that will work for the entire enterprise because that's its action is called an enterprise functions for example you take customer entity and you talk to finance you talk to say all the top customer customer care each of those


Department look at the customer differently in finance they look at did he pay his bill and suffocated in sales they look at good services is he using can we upsell cross-sell upgraded in customer care they are looking at does have problems does he have problems in the past what kind of problems etc so the models are not different and what you need to do is you need to establish a boundary in which your model is unique and that is basically in one domain we will could have one model that is unique and then when you talk to another team or


whatever the physic designing another domain you need to establish what the boundaries are exact like and what is important how to transform and connect your model with their model it's really a good concept I haven't talk any more about it but you should look at it really a lot of materials online about it so the next topic when it comes to micro services actually this is one of my favorites and also a lot of debate is around debt when it comes to the communication patterns between micro services how do they


communicate with each other there's actually two big patterns one is choreography the other is orchestration choreography you have micro services that are independent they function as let's say individuals with a set of rules in which they talk to each other but there is no central micro service or whatever that is orchestrating them in orchestration you have a central entity that is orchestrate that is telling the other micro services what to do which one is better neither because it depends on the use case that you have


mostly in whatever you are designing NIT there is no one-size-fits-all you need to be open-minded and you need to look at exact use case that you have so for example a choreography is a really great it is actually my preferred method because in choreography the main benefit is that these micro services are completely decoupled when one goes down it doesn't influence the rest if you need to remove one micro service and implement it in using a different technology or whatever you can do it because they communicate asynchronously


they have a set of interfaces that are standardized and this really gives you a lot of flexibility but what is a problem with choreography is that if you have a complex system or a complex process that you need to implement it's really hard to visual the communication so if you can't visual that which visual visual visualize that communication don't use choreography you can visuals visualize it by using open tracing and technologies like that there there are ways but if you can't do it don't want choreography orchestration on the other


part is great for example when I use it in an order execution process when you have a process that is sequential and you really need to follow the exact amount of steps that are needed to talking for example in telco when you do order execution you need to create a customer you need to create his assets you need to send his equipment to the delivery service you need to talk to other telephone operators regarding number transfer and stuff like that you need to activate the subscription for the customer in billing


there's a lot of activities that you need to do all of those can be implemented as different micro-services but many work is straight you are really sure that everything is happening the very data truth but you shouldn't do orchestration in code because the same is choreography you won't have any visibility of that when you do orchestration my advice is use a BPM tool if a really good BPM tools on the market that you can use and they intend visibility in the control the process is great so just to show you this is one BPM tool and


this is how it how it looks like you have where are the process instances some are in incidence some are not so it's a it's really a good way to see what is happening in that process you really do if something is amiss in incident you can easily see what's happening so the level of control here is really great so that's my advice when it comes to orchestration but the most important part is no one-size-fits-all so let's move on just some quick best practices when it comes to micro-services design actually the only


don't here is don't put everything in code use the technologies that are on the market and that are really great for example if you have a lot of business rules that you need to implement there are great rule engines that are fast and that you can use instead of writing millions of ifs and cases in your code the only trick here is if you're losing in a row if you are using using and there are there two to two ways you can use the rule and you can embed it in your code and then it's very to just use the engine and edit code or you can


build it as a independent standalone service but if you do that it's important that it's a passed rule that the rule engine only executes rules it doesn't query any database or back-end or anything for data it should look something like that this is request this is response you give in the the rule that it needs to execute you give him the data and then it executes the rule inhibit the response back and it should be in memory and it should be really fast as you see here it should really be a number of milliseconds the seconds that you can


count on your fingers and toes because if it's slower than that you can forget about any kind of performance that uses the rule engine so it's really important that it's fast proof it's in memory and it doesn't query anything it's just an engine also use artifact repositories they're really easy to use really beneficial use tools for logging don't log to file system database and then if you have an incident spend three hours wearing debt use pools the terror in on the market protect the self is for monitoring use time series databases


because they can handle a lot of data and they have great visualization tools that you can look at in the operations part so I'm not going into specific technologies that you can use but for all of these technologies you have open source solutions if you want to know I can tell you what we are using but maybe later on or in the question space and the last but probably the most important things he is work together with business because a micro service or any softer that is its own purpose doesn't make sense so you should you should build it


together with people from business and you should build stuff that really solves some business issue that can be some new capabilities of extra functionalities they can be just that you are doing refactoring transform suddenly some old implementation into micro service because you want better performance which will result in better customer experience these are all valid reasons but don't do it without talking to business and having really good memories because basically micro service should be analog to a business domain


and it should implement business capabilities so that is really an important part here so they could actually be the first step when you're designing the micro-service know what is the challenge and what are the capabilities from the business side that you need to solve and then go to the technical part and that's it about the micro services architecture I will now give the work to even we will talk more about integration about integration patterns so when we were doing this kind of migrations basically integration


become as you move to micro service integration becomes more of a hot topic than it was before so some of our experiences don't use multi layer of API clothes what I mean by this we have a situation currently were fronting is going bacon that is calling second packet that's calling terror that comes forth bacon so basically you have this connection of flows where each layer is introducing their own bottlenecks there are slowdowns so don't do this second is what Luke also mentioned in micro service architecture try to do the


main driven design in telco industry where we are happy that we have a global guidelines regarding this there is a community gathered around TM for telemedicine forum organization that already built for us some models so we know what are the customers what are the products what are the services resources payments so we already have domain driven design that can be followed basically we already have domain prepared that just need adjustment for a specific tau and this domain is extensive enough so that you can extend it without let's say many


travels of course no brainers don't use non scalable or low performance by non-scalable I mean try to not reach the limit where you need to pay for extra licenses or where you are on all the technology that you cannot horizontally scale I mean you can always add more CPU power but this is something that needs to be avoided and also try to use the try enough to use no documented API I know we have cases where there's an API working for 10 years it works good it's just that developers that did that long time goal of the company and you


don't want to rely on such API because when the problem starts then you will need to redesign the API in a matter of hours just for it to work so if you have something that is old and I mean how could we have a lot of old technologies that are there because they work and we have systems that are old 10 or more years you have stuff that is not documented so what to do in integration patterns try to use data provider API so if you cannot come close to the data by varying the database try to come to the layer that is closest to that database


layer in some cases you can of the show some on the shelf solution where it is a black box you only have the API provided by the solution no one is telling you ok in the transformation let's replace it you purchased if you want to have return of investment over it and basically you should use it but you don't want to use first API that is going second API that is calling this code safety I try to consume it directly and then use rule engine to apply any rules that you have over data to provide domain-driven the main data that these


needs for that particular purpose and try to do something that is a try to ruin something that is easily upgradeable to new principles so if you have old APR don't let you see okay you need to spend a little more effort to be compliant with the new proposals new strategy invest in that API clean it out and upgraded don't built everything from scratch but of course when you have 10 or 20 years of development a lot of things are piled around so sometimes you just need to have a clean cut and do it from the beginning


regarding the integration patterns we found let's say something like five turns I will present some of here and what is important is that one pattern doesn't fit all so first when we are trying to figure out what betters we first want split it up by CQRS principle when we looked at our logs and how users are accessing our data most of the data access is read access so a lot of people on the self-care device is going to see what is the state of their consumption are they paid have they do the bills and stuff like this so most of


the data is coming from down code to the customer and then then you have a payment of bills activating some new adam strange's tariff and this is the smaller amount of command like queries so basically one pattern will not feel cold try to separate your integration patterns in your API by CQRS principle do stuff enhancing if if you have problems with executing the commands give the user the info okay it is your request has been received and do it quickly and then we're on if you have five ten or 20 seconds where you need to process this


data is Luka mentioned our orders are complex sometimes we go from creating the customer to checking the dista to see if the speed is good enough so that he can receive some HD channels it takes some time so try to do it innocent way and try to keep it simple so these are some of the patterns I will share with you so this is the CDC of data replication better so we have legacy beckons legacy API what we are trying to do is copy the data from these databases to our application database and then do a data transformation from legacy model


to new the main model so in legacy VIII you often have situations where in some applications there is a mix up of many models we have for example customer data and with five different databases so this shouldn't be the case in the domain driven design so what we are trying to do is pull this data as fast as possible through the data replication layer and then transform this data to a domain driven date and then when fronton describing your microservice you have data already prepared in the model that is by the domain driven design and what


is important here data replication it's not only a CDC so depending on how and is the data change how often users needs to be aware of this data change it can be a job it can be a night and drop so basically it's up to you to see what fits your purpose and how you do the replication CDC he's only one of the options that that is applicable the second pattern that we saw is when you have off-the-shelf solution or where you need to consume something not directly from the database so how we did it for example in our case


this is the users consumption API that is connected to a different billing systems and this data exchange really really frequently CDC and the amount of data that is coming through through the mediation system it's really big so CDC would just introduce the override that is not good enough so what we did first we simplify the architecture currently we have these multi layer API flows that we're introducing their timelines and bottlenecks now we connected either directly to the database or where we had all the shells who should be connected


to a soft services that that of the shell solution presented and then we are doing the query this solutions transforming it in specific service layer and then presenting it to the customers of course I put the cash here because usually users don't want to see every second but what their data consumption so you can put this kind of data in a cache for 10 or 50 minutes 10 minutes should be enough for users to buy safe if he wants to return to the to the consumption data he can check it out but also need to be aware of you need to


sometimes give him a pull to refresh capability if he really wants to see data exactly this time you can do it by demand but most of the users just when login to the South Korea they want to see the current consumption and if they jump over to the details you already have this data cache you can you can show it to them okay and the third layer I was mentioning regarding the commands that are being executed basically if you have situation where we are executing for example changing the death of a customer and this takes some time


because a process needs to be run you need to connect to all those legacy systems that are responsible for changing the tariff from the what's a BSS layer to OSS there sometimes this takes time so trying to see what are the transactions that you have minimal number number of errors so if you have a change start process that is really if qualification of well staff user can jump into is really good at first then your execution part will mostly be successful but if it's slow you can use the pattern where you store the message


you tell immediately to the user your request has been stored and then you start this in a synchronous way executing this message when this is successful send a push notification to the user so we have self care applications we have SMS these we can send to the user now your type has been activated so try to use whether you see that you have a really higher percentage of successful executions trying to do this in an ASIC way okay so other things regarding data aggregation and data consolidation don't do it on demand so


this is more let's say applicable to beckoned for fronting players or front-end closer api's don't try to do on the on demand or on fly aggregation to Mapes try to set up a domain and prepare it how how easy it is fitting don't try to do aggregation over many domains it really means that you did something wrong if you need metrics if you need aggregated metrics try to prepare it in advance by relying on event-driven messages so instead of querying the database to see how much SIM cards customer has you can do once he is activating a new


SIM card or something like this put this invent and store it somewhere in the bucket and then when you want to know how much SIM cards because queried bucket don't go to the database and try to query select counter from now number to return number of SIM cards also if you have situations where some of the micro services are sharing the same database but in a different schema don't try to consolidate data on the database level database should be treated as private fields so they are owned by microcircuits and there is a layer of a


business logic above this data where perhaps when you are querying Sunday you won't receive a correct response so if you need to aggregate something prepare in advance do it in advance and don't do it on the fly especially directly on the database later it is okay as this is done by data where perhaps this is their job but I'm speaking about direct confirmation of data from digital channels from self-corrects also regarding the micro service approach and api's try to do some sort of management this really makes things easier so you


can free API management tools you have a really expensive one so depending on your budget try to fit something but it is really important that over your api's you know who is technical and business owner you have the governance you know when new version of API will be set up you know when old version will be retired you will know who are in some cases we needed to introduce throttling to find out who are our consumers because some people we didn't even know we're consuming the PLC cool procedures started complaining about


timeouts so API management where you know what is the current version of API where we know who is consuming it and who is the owner of it this is a way to go don't treat ideas only as one protocol API so this is what is the domain and layer separation the main separation is domain driven design separation and layer separation is try to keep things controller service and repository to keep them out of each other way so every layer has its own boundary and the boundary shouldn't be crossed if you manage to do this then it's not a


problem for you to have an API that has repository and service and then it has controllers towards rest towards GRP see towards the Kafka or other streaming platforms so try to separate it both horizontally and vertically clean architecture programming the development practices so I try to split them up by organization level and T blow so in order to have a good development and good api's some rules in the organization needs to be set up it talking in small companies when you have a small team and startup you are


creating your own rules but on organization level you need to have support from operations department commits have support from DBA from system department so some rules needs to be set up what you don't want to do is to have multiple text acts try to find the text type that is okay for you and try to stick out with it you don't want to get yourself in a situation where some developer developed micro services in I don't know Godwin ergo because it was really right technology for him at that time and that feel of the company cool


govern disk so try to have a tech stack that is let's say a future proof and that you see it is really something used in beckons on beckons where let's say lucky here we are not as volatile eyes for example JavaScript frameworks became Java before we have Java now just we have different flavors but this is it also try not to limit the tech stack in a way that I don't know if you need something where you see graph databases will be good don't be limited by ok do it in the positives or something like this don't do like I say tech


development I mean this is no transformation you are trying to move your team also from all these new technologies and you have a bunch of experts on all technology and their juniors in new technology and sometimes when you have deadlines the decision will be you we can do it in 30 minutes in all technology or in 3 hours in new technology invest in new first time it will be three hours next time it will be two and a half I mean no one no one no one learns King in 30 minutes so you need experience you don't want to go


back because then you achieve nothing don't skip continuous integration I know that for enterprises such as ours continuous delivery where you with your coat production is usually a no-go because we want to help human you need acceptance tests because we want to have deployments to production over several systems that needs to be synchronized so it's ok to split CI and CB but don't skip CI so caliber books that has minimum CI minimum CI at least that is running automatically code review in unit tests and whatever


tests you put in the code don't do a constant move forward so we also can experience where new user stories are just arriving arriving arriving and you didn't manage to have time for optimization I don't mean just a small organization when you are solving some visible technical depth where you put something to commit to do and then you find some time in the next sprint to fix it I'm talking about getting the overall picture seen so as developer I learn everyday I find some blog I find some new technology so because of the move forward so six


months from now I will be smarter regarding some things that I am current and kind of writing some code so six months later perhaps I I need some Sprint's to sit down to the overall picture and to see okay what can be optimized what I learned in the last six months that can help me move further otherwise you will just have let's say cloak that all parts are working but it's still showing the wrong time and try to cover clear member robz try to have a governance that is known when exactly something in is expected


developers like to know when they have meetings with their product owners they want to know up until when the user story will be groomed and finalized but they can start development so scrum is a really good principle here because you can set up the deadlines you can set up when it's expected what is expected and what amount of work needs to be done in the next five or ten days so on the organization level try to stick with these principles try to have this in a dedicated time slows so that developers can plan this is the greatest thing that


can be done when you have the sprint and when you have the agile approach on the team level you want to have guidelines especially when you're doing digital transformation for something new you need to create the guidelines you need to tell either to let's say new technology junior developers course and many are experts that you have that they which where they don't know the telcos tech you want to have for them the guidelines how things are built so for example we have our horizontal split finger our own products where it is not what


each product does and what kind of code should be in some kind of credit so this is a guideline that is really helpful to a team don't prioritize that line over tests this is easier said than done but what I would suggest is and never communicated to business how much time you need for writing only code when you're writing a code without tests you don't write code you you write you're writing a wake-up call into a by the gracious theme that something is not working product so you are the ones that will be cold when something is not


working not your stakeholder and in my experiences older you get you are much let's say much less open towards this kind of course try to do code reviews and try to establish recording practices so this is basically the constant work so you need to on the team level to help retrospective to gather feedback from your team what was good what was not good this is the only way how team will work on and you need to also do a knowledge transfer so basically it's not just knowledge transfer to a new team members it's not just transfer when you


found something you race and blow we found something that can Ament assertions you will do search J or compass so basically when you find something that will speed up the team you you hear some R&D priming sprint because this is how you move forward of also regarding the library dependency management it is okay to make a test out but you will run into problems where you have one library in Springwood - you have second in two lows one to live - so when they when you start connecting them together things will break out because there will be


resistance is so to try to create a project that will have your Bill of Materials of dependencies and then try to agree okay let's use this for next portal and then after three months we will see what has been upgraded and then move forward any pressure let's say a bit slower here because they are not always let's say on the latest on the edge technology but this can also be good because you can check you can leave others to solve the initial bugs that new version of technology growth so you can be much more let's say seven safe


side here also what I wanted to want to show is basically do the unit tests they are really helpful when you are moving forward and you are doing this dependency upgrade if you run your unit tests when jumping from java to java and everything us it will be sure that okay this can work so this is let's say a really important thing to do so this is it from from my side post we are now for any questions that you have see there is only one question regarding two videos okay and so I'm back I'm back with you guys


okay let me let me publish question number one will there will those videos be available afterwards yes they will if the question refers to rid of this session yes video will be available at the developers website and that bug website so it may be two days let's see another another question do you recommend using graph QL guys yeah great I think I like it a lot but for front-end to me that beckoned forefront and better and then there is you should use graph QL and it really makes sense especially when it comes to performance


but when it comes to you know internal our integration between various systems in the landscape there are some other protocols like GRDC or rest like revenues but for front back in the Disco as close to the front as possible but when you are providing domain driven design especially when you are an engine present you need to have authorization concept and who can see what and in this case is for us rest or other let's say different communication or district GRP C or even driven are better choice because you have stabilized model and


you know what to expect and what to return and then over this you can easily provide authorization rules what needs to be removed or pre-checked and Jeff Kurr can be a part of BFF that can then query the main driven micro services yeah yeah I would say that graphic is definitely a future-proof fronting because you look at a lot of VI that expose the front and the mostly have too much data so they are big and clumsy and have a great old formance or they don't have enough data so you have to carry several api's which is a even worse than


you have a lot of network latency so yeah forefront indefinitely I think that is the future but we have another question regarding grant how do we handle graphically on with microservices know the kind of severe let's say focused on the banking services as we are trying to provide clinical if you have spec sheet there I know that it's a moment of time I piloted I think it was a framework that had this kind of support on micro service though where it can consume a different micro services and then do the graph QL processing go


read some of the caps you can check that framework to see if it works for you we are not using ok Mario is asking how larger the teams and how do you break down big organizational goals to team product stories ok definitely actually the Arita entire company is now let's say in an age of transformation we went from a standard way of doing project which was basically waterfall you know you have marketing that thinks of an idea then do a specification it goes 2080 implements now a transferring to the agile way of working where we have sort


of a matrix we have agile tribes that are in charge of larger domain with the squats inside that they charge of smaller domains so that's what we're doing on the level of the entire organization and when we connect it to the architecture what we want to do is have one squad in charge of one micro service that's that's our goal so the tribes handle the backlog when it comes to functionality capabilities that stock we want to do when it comes to develop development use force that are in charge of each squad instead of one micro service and that's


how we want to achieve that but we are in the setup phase okay Marty McFly is asking can you share your software stack which databases are you using basically I put stack on the one of the slides we are using for some cases yeah okay if you have a lot of Oracle databases in our legacy stack but for all new stuffs we are using all the center also databases okay what approach you use for performance and integration tests in micro services distributed architecture so we are relying on most here regarding the integration test so of course I


mentioned that we always have user acceptance testing that is done by testing team but when you are developing for us a great Mowgli's word test containers where you could mock a database and also a wire because most of our integrations are based on I just saw or PLC code but this cannot replace the real end-to-end testing but it can speed up development because you don't need to be let's say you don't need to create test users fitting for some of the needs that you want to test you can just set up a test container with test


database and then put some your own test data values or set up a variable that is returning what's officers should return okay so Mario is asking how do you do public API is in micro service environment does every team own their own public API or is there some common part currently we don't work most of our API are punished either to our South Carroll occasions or to our business partners so this is something where we don't have much experience between Luca mentioned how we are owning the ice but they're usually


consumed by our own applications okay so you said spraying boot-up doesn't equal micro service what technologies do you suggest to use we are using Java spin boot actually it's not enough just to implement an application Java string boot to call the macro service to implement a micro service in Java spin would you need to have a Java string motive you need to have a database with data you have to expose API on that micro service and that micro service should encamp encapsulate one specific domain that is connected to business domain


that is automatic services so feel free to use Java spin booty reuse it also but it's just that a lot of times people build Java spin boot applications expose an API but they have no data the Java spring boot application and there is a lot of different systems for data and this is not a market service this is a proxy you can you can implement it whatever technology but micro service should know on its own date that's what disk I mean we also have the pattern that is not gone into data because as I mentioned we have the


technologies or systems that are purchased the billing system mediation system so you don't want to say now ok let's draw billing out of the portfolio and do our own billing you don't do that but if you are consuming data from be linked and you want to create micro service that is not real micro service bye-bye we can officially path you want to go as closest to the data as possible and then provide your own business rules again business rules exposed to our standard business rule solution and then expose this to the outer layer but if


you just create a spring boot that that is consuming some legacy API that is going God knows where then you have that multi layer flow that you really want to avoid yeah and you are not scalable it doesn't matter how many micro services you deploy because it still where is the one single database for example billing system for data so that should get your bottleneck and single point of failure if that goes down all of the micro services doesn't matter if you have one more connect so in order to build really scalable like service has to have its


own date but as even says if you clearly can't do that every time so the those patterns were you connect to legacy systems but connect as little layers as possible ok so we have our last question coming from Portugal hello from Portugal great talk how do you deal with scalability of software from vendors or do you avoid using software that are behind licensing okay he was vendor software definitely and we use some license after all so you know Intel code there is really a lot of domains and for some you have a


excellent excellent out-of-the-box vendor software that you can use and it really doesn't make sense to you know invent the wheel and implement their own billing yeah yeah you know implement 20 micro services that does not tell Kabila for example then you should use you use a vendor solution you paid the license and when it comes to scalability those softer scalable you know you can create a cluster deploy so several instances and that's let's say well no one approach how you can make monolith soft who is consuming and how much today so


for example users consumption API that I mention it is connected to a billing system that this purchase con solution so if you can query data from billing then put in cash or even a long term cash because you see that this data won't be changed regularly for example we have a say is containing god bills to baby so if you once query the pay bill you know the amount you know when it was paid you can put it in cash if user comes inquiries he must to see his last 12 bills or some aggregations I don't know average bill amount buried for last


12 months you don't need to query yes it be you can have it in your cash and query the cash or in memory store in this case cash is more like a memory storage and so it's dependent scenarios you try to find how data is used and how often data is changed on the back end and then you made a wrapper around that that can make your life easier where you don't need to go directly to the back and then you don't need to scare it that much okay a lot of questions today thank you Luca thank you even for your time [Music]


okay I'm sure many of you have already seen this because it was rather widely announced but the open AI team has announced a new model that produces pictures instead of text so as you can see right here on the Left you'll always see like a half a picture and on the right is the ground truth so they took this picture they simply cut the bottom half right here and then they let the model sort of imagine what they cut away and what it comes up with is pretty cool I have to say like look at the birds like this is

Image GPT: Generative Pretraining from Pixels (Paper Explained)


just awesome but the special thing about this isn't that it simply completes pictures the special thing about it is it does it one pixel by pixel so basically it goes at this pixel right here and asks ok what's that pixel and then what's that pixel and then what's that pixel and so on so it is basically a like a language model but four pixels in that it goes over the images in order basically like this or like always from left to right left to right left to right and it has no clue of the spatial relations between the pixels it needs to


learn that by itself as opposed to a convolutional neural network which is specifically designed such that if you want to predict this pixel right here then it's specifically designed to say ok the most important information is probably around that pixel and then some like other important information is while around that pixel so cnn's are built with this in mind whereas this model right here which is also known as image GPT isn't doesn't have any of that it's simply a transformer model that goes over these pixels one by one and we'll see how


that's done there are some more examples right here particularly cool is the cat and you see that there is the beginning of this little white thing here which is this card and the completions of the model yes very interesting the model can of course as a language model can also sample by itself just random images you sample them once through and this is what it comes up with so these are pretty good quality images for a model that just produces one pixel by one pixel now this idea of one pixel by pixel isn't new this has


been around before but the investigation here is basically how much can we how far can we push these generative models for pre-training hi there this is Yannick from post-production I've realized that I've forgotten to even read the name of the paper so it's called generative pre-training from pixels by March and Alec Radford round child jeff whoo he won't jus profile a dairy wall David Lewin and Ilya sutskever and since Henry AI labs has already made a video on this this video is going to be more of kind of a rumble


rant about what I find interesting about the paper and some thoughts about it rather than like a classic explanation I hope you still enjoy that so what you saw on the right wasn't even though this isn't the final result the supposed result this is simply the pre-training task it's fun to look at it but the actual object objective of the paper is the following what if we train we pre train on a large data set to generate work good images like these or we to complete images like these and then we fine-tune on a


classification task and the answer is here they say on C 410 we achieve 60 96.3% accuracy with a linear probe outperforming a super wide supervised the wide ResNet and the 99 cent accuracy with full fine tuning matching the top supervisor pre-trained models an even larger model trained on a mixture of imagenet and web images is competitive with self supervised benchmarks on image net achieving 72 top one accuracy on a linear probe of our features so the goal here is that you have a data set that you want to trend


like train a classifier on so usually you have a data set and the data set has images and you put them through like a convolutional neural network and then you have to classify the image into one of I don't know how many classes on C for ten that's ten classes on image and it's a thousand and the data set is these images together with these labels now the idea of pre training is that you some where have a bigger data set that is sort of similar to the small data set but it's similar enough such that the network could learn something so what


you want to do first is your first one it take the large data set terrain this network right here and then in a second step fine-tune the network on this smaller data set and you sort of hope that what you learned from the large data set right here transfers over a little bit of knowledge you already have a little bit of knowledge and you can make better use of the data that you have right here now the question is how do you do this pre training and of course this has a long tradition well long for maybe two or three years right


now in the language community where people they pre trained these large models like we've just seen GP t3 or Bert was one of them they pre trained these large Transformer models on text and then to fine-tune them on classification tasks for text and that's what this paper is doing right here they pre trained a transformer that is a GPT to scale model they pre train it on image generation and then they fine-tune it or transfer learn it to classification tasks and the point of the papers to say that like in text data in text data we have made


pretty good pretty good experiences with doing this with pre-training a generative model and then fine-tuning on a classification task while so far in images all we've ever done is we've pre-trained this pre training task he usually is a classification task or like a self supervised task with a contrastive loss or something like this what they're doing new is the generative modeling in the pre as a pre training and again this isn't like entirely new but they show that if you throw a lot of computers at it and lots of data and a


big model then that can work equally well to these self supervised tasks so their model as I said is pretty pretty simple they take an image and they unrolled the image now an fully unrolled image on let's say image net has 224 squared pixels and that times three right because you have three color channels that's too large even for an open a supercomputer so what they do is first they down scale the image so they down scale it's not as drastic as here where you just get a three by three image but they do down scale it to like


a 32 by 32 or a 64 by 64 then they unroll it which simply means they go through the image like this and make a sequence out of it because their models are naturally made for text sequences they simply put the image into a text sequence they further simplify this by reducing the three colour channels to a single one so they have their own color representation and basically yeah they reduce the three colour channels to one channel that simply indexes the color in their color representation and they say still pretty good it's pretty faithful


so ultimately they end up with like 32 squared length representation of their image and then they do one of two things they either do auto regressive generative pre-training which is the sort of GPT - style pre-training and the the idea here is that you always want to predict the next pixel of a sequence so you can see right here that's the sequence that you are sorry that's the sequence that you input and you always want to predict what is the next pixel and in this case you've seen you see that we've already predicted everything


here we've already predicted everything up to this red pixel so you want to know what's this next pixel this thing right here what's this going to be and the diagram here basically shows you how the attention flow so every position in this transformer and if you don't know what a transformer is I haven't made a video about attention is all you need where these are explained but briefly every position here can sort of send information can send information only in one direction as to so you train all of these in parallel and when you predict


this pixel right here you only want information from whatever it was before that pixel otherwise the model could cheat right otherwise the model could simply learn to copy over the value but the attention pattern here is simply to show you that this is auto regressive and it's in one direction so you always want to predict the next pixel and then from all of this you want to predict the next pixel and from all of this you want to predict the next pixel this is in contrast to this objective here that comes from Bert and


I've also made a video on Bert what you do in Bert is you simply take that image and you cross a block out two of the pixels or many of the pixels and you simply ask your network to reconstruct those pixels okay and now you can see the attention flows in all direction birth the B stands actually for a bi-directional so this is the contrast to the autoregressive pre training framework now the these two things have been applied in text both the autoregressive is usually it's easier to actually make it produce something like


we saw producing these images because you can always just predict the next pixel and then the next and then the next and then the next whereas in Beart it's a bit more unclear how you would produce things in a consistent manner because the predictions of these two pixels right here they are independent it's one forward pass and then both of these are predicted but other papers have tried to solve this like this not excel net I forget I forget its name it's something with an X and yeah but but these are the two objectives they look at and it turns


out they sort of trade off a bit they work equally well or a bit better and a bit worse depending on the task so once they have done this so they simply feed images and you will notice that you don't need any labels for this so what you'll do is simply input an image and then simply take away half of it like this and then predict that pixel and then you want to predict that pixel and then you want to predict that pixel right that's all like you do with text and invert you simply input an image cross out pixels and then predict them


so you don't need labels for this and that's why you can do it with this big data set and you can do it in an unsupervised fashion so you can just crawl the internet for four images and just feed this in into there and it will sort of learn to produce these images now the question is if you produce if you learn to produce these images does that help you for classification and there they have two methods of of assessing this the bottom one here is the fine-tuning method where you simply so this is supposed to be the


representation you learn in the different layers of the network so this is supposed to be this thing right here what you'll do is you'll simply fine-tune that means you on top of this representation you add a classification head that has two outputs cat or dog and you train this entire network on your small data set that we discussed before so you train the entire network all of the parameters this is called fine tuning in contrast to that what you can do is you can simply and this is the easy way you can simply add this


classification head with two outputs and then only train this classification head and that is won't perform as well but it gives you sort of a better idea of how good is the representation that this network right here learned and on top of that so if you spin this idea further you can actually go and do this at any intermediate layer right here so you can forward propagate until layer two right here and then here you add your classification head into the two into the two classes and you only train the classification head that being said you


can also do this with fine tuning but in this case this is called a linear probe and it is often used to assess how good the a representation in intermediate layers is whereas what it actually does is assessing how linearly classifiable a representation is which isn't the same as how useful or how informative but it is one way to to assess these things okay so these are the two things they assess alright so for as for data sets for C 410 they use like C for 10 and c for 100 as data sets and the STL 10 and there you have to keep in mind the pre


training is done on imagenet for those so that you pre train on imagenet without the labels and then you transfer learn or fine tune or or linear probe on these small data sets whereas later we're going to look at image net and they're the pre-training as I understand it is done on image net itself but also a wider collection of a hundred million or so images from the web from the internet okay so as you can see right here this is what happens if you do this linear probing and you can see it works pretty well so you get like


a ninety-five ninety-six percent accuracy with linear probes this is very powerful so it's not easy to get 96 percent on C for ten I mean current state of the art is like ninety nine percent but still 96 percent is pretty good and this is the so the entire network there is this big giant network that you input your image into and then there is this one linear layer that does the classification and all of this right here has not been trained with classification in mind it simply has been trained to reproduce images it


hasn't even been trained on C for ten as far as I understand has been trained on image net so the the this is to stress how cool or how significant this result is basically that just a linear probe on top of that will give you such a good accuracy and the second thing that is obvious right here is this bottom axis is the layer so this is the layer where they attach the linear probe and usually if you pre train a network with a classification task in mind so you pre train it with the labels or maybe even without the labels in a self supervised


the way or something like this usually the last layer has the best representation for classification but here the special thing is that the intermediate layers in the middle have the best representation you can see that representation quality in terms of linear probing falls off as they sort of it falls off as they go into higher layers and this is consistent across the datasets as you can see and the the idea here is or the way they interpret it is that if you have an image right here Dada Dada Dada and they you've blocked part of it


so you've blocked this and this wrong way around this so you've generated everything and now your task is to predict the next pixel right so you're you train to predict this next pixel right here and the idea is that as you put the image through the network what it will do is sort of since the first layers they're going to be if you're going to be similar to a CNN they're going to be doing some low level of feature transformation thing right but also the last layers they're going to really care about what's the exact


pixel that goes here right since it's their job to to do that they're going to care what color does it need to have you know what exact luminosity and so on how does it fit in with the previous pixels and so on where as so that's that's also good but it's not just low level information and consistency with other pixels or something like this at some point if you want to generate consistent images and we saw that this model can generate consistent images at some point there needs to be some kind of a notion of the global information in the picture


right such because the images are consistent throughout so there needs to be some some notion of what is in that image as a whole and that's the exact information that we need for classification and the only way that could actually be is here in the middle since you know that's the place so the hypothesis is that the these models somehow learn a higher-level representation of global information somewhere in the middle before they then specify that information again down to predict the actual pixel and that's why


the best representations for classification are in the middle so this is one of the this is actually the interesting finding or one of the interesting findings of this paper means cool that they can reach a good accuracy but to recognize that maybe in these these generative models they have some intermediate stage where they represent the global information and that will actually make the best representation okay the second cool thing right here is that you can see they have different sizes of models so the IGP TL I believe


is something like sixty layers then this is like 48 layers and this is 32 layers we don't really so these are on the olive on the scale of GPT to either a little bigger or a little smaller it's not like a GPT three scale where you need a ginormous supercomputer though they do do a lot of computation but it this still sort of fits within hardware of a standard size and not like exascale what's interesting right here is that you can see the larger models they reach a lower validation loss so here is the validation loss larger model if you


train them on so these checkpoints here are always after the same amount of steps the larger models do reach a lower validation loss right here as you can see so this is the large this is the medium this is the small and also you can see that on this axis is the linear probe accuracy so this is whenever you you go and you find the best intermediate layer for linear probing you probe it and you record the accuracy so you can see a general trend as your validation loss goes down the linear probe accuracy goes up so there


is a connection like it is in text models and text models there's a connection of the perplexity of your language model and the quality that of the representation you get for downstream tasks in this model it seems to be the exact same thing there is a connection between reaching lower validation laws and reaching a higher performance on classification so that's one interesting thing the general trend - up to the upper right corner the other interesting and even arguably even more interesting thing is that if you look at


the same validation loss so at this point all of these models have the same validation all's yet still the bigger model is better right you can see right here the bigger model outperforms the smaller model even though they have the same validation loss on the image modeling task and this is also something that openly I in their in their text papers has stressed that the larger models they seem to be somehow more capable of forming good representations even you know even if they have the same loss so again this this could just be


sort of a training data remember training data remembering thing and when I set that in GPT three I didn't actually mean explicit remembering of training data I meant a kind of a fuzzy remembering of training data of I formulate that in the in the comments but I I feel a lot of people have misunderstood me there here I think it's a much harder harder to estimate what's going on also since image pixels humans don't have a super good model of an on image pixels in their head as we have about text as you can see if you then fine tune so for now


we've just do linear probing if you fine-tune these architectures then you reach like a 99% accuracy on C for ten which is on par with the best models that we have so G pipe is supervised pre trained on imagenet but also I guess uses a bunch of data element ation while these image GPT it uses minimal data augmentation I think they simply random crop a little bit and that's about it so they also experiment around with this Bert objective so until now this was all the this was all this autoregressive objective and I feel the open hi people


are a bit more of a fan of the autoregressive objective just given what they've done so far in their papers and you can see here comparison of the two objectives on C for 10 and on image net again C for 10 is pre trained with image net and image net itself is pre trained with like a larger collection of images from the web all the pre training is done without labels now the blue is what you can reach with a linear probe and the orange is then on top of that what you can reach by fine-tuning okay so no linear profile tuning oh I have to say


that the fine tuning is always done at the end so even though the linear probe even though the linear probe can be attached anywhere in between and it's often useful to do that as we saw because the in between layers are the best they say they tried fine-tuning also in from in between but it always worked out best whenever you fine-tune whenever you fine-tune you take actually the last layer so that kind of gives you an idea that the model is then it's sort of what seems to be important is this coming up with the higher level representation and


then once you fine tune you're probably able to push that representation through to the end because of your training signal but if you hadn't done the pre-training you wouldn't even have that higher level representation and then the signal I guess is not strong enough to back propagate through the whole model it would be very interesting if they investigate if they do this linear probe analysis again after they fine-tune the model that and to see if then still it is the intermediate layers that have the best representation or if now the best


representation in a linear probe sense shifted towards the end I'm gonna guess it's shifted towards the end but I sort of want to even see if the accuracy of the linear probe in the middle does it keep the same right so does the curve go like this this is the linear probe when you simply pre-trained right this is linear probe accuracy the question would be does it change to be like this or does it change to be like this this is supposed to be the same at the end so basically does it stay as good as it is but simply get better at


the end or does the representation like in this curve does the good representation now shift towards the end and leave the lower layer with even more capacity to do some low level stuff yeah maybe they've done this I haven't seen it so and as you can see these Bert and autoregressive object if they sort of trade off so the birthda tends to do poorly in the linear probe setting but then it catches up during fine tuning in C for 10 almost being at the level of the autoregressive and in an image net actually outperforming it this this


darker thing here it simply means that you a ver across different maskings of Bert because I guess even in classification it's not entirely clear how to get a signal out of Bert because they don't do this CLS vector with Bert what they do for classification and linear probing and it's written up here they simply take the they do an average pooling I think they do an average pooling of the of all the representations of the sequence and the last thing that I've also forgotten there's a lot of stuff when they fine


tune while fine-tuning while fine-tuning the classification loss yields reasonable downstream performance we find empirically that the joint objective the generative objective and the classification objective works even better okay so even when you fine-tune with this model you have to keep the generative modeling part the generative loss around and then it performs even more better more well whatever that word is so that's also something to think about I think this this paper right here it kind of lays down a lot of cool


things that you can think about and it gives rise to a lot of hypotheses of how does this stuff work why does this stuff work I don't even think that the the numbers are the most important thing it's mostly the fact of the effects and what does it mean okay so this was my take on it it was it's more kind of a my rant of what I find special about this paper then about the actual paper you can look at the paper their numbers are pretty good on imagenet they do not reach the same like SuperDuper performance as they do on c 410 and i


guess that's probably because they have to downscale the image net images way more than they have to downscale to see 410 images because those are of course only 32 by 32 so because they have to downscale so much they lose probably a lot of information and I would be interested to see if there is a way to involve convolution in this in all of this so to do the downscaling that in a learned manner with convolutions or something I'm sure this has all been done already I'm just lazy to look it up yeah so I invite you


convolution is a measure of overlap between two functions as one slides over the other mathematically it's a sum of products the standard convolution operation is slow to perform however we can speed this up with an alternative method that is the topic of this video depth wise separable convolution let's first very quickly go over the basics of convolution on an input volume consider an input volume f.o shape d f cross d f cross m where DF is the width and height of the input volume and M is the number of input channels if a color image was

Depthwise Separable Convolution - A FASTER CONVOLUTION!


an input then M would be equal to 3/4 the RG and B channels we apply convolution on a kernel K of shape DK cross DK cross M this will give us an output of shape D G cross DG cross 1 if we apply n such kernels on the input then we get an output volume G of shape DG cross DG cross n the convolution operation takes the sum of products of the input and the kernel to return a scalar this operation is continued by sliding the kernel over the input I've explained this concept in detail on my video on convolution neural networks


check that out for a clear understanding I'm more concerned now with the cost of this convolution operation so let's take a look at that we can measure the computation required for convolution by taking a look at the number of multiplications required so why is that it's because multiplication is an expensive operation relative to addition so let's determine the number of multiplications for one convolution operation the number of multiplications is the number of elements in that kernel so that would be D K times D K times M


multiplications but we slide this kernel over the input we perform DG convolutions along the width and DG convolutions along the height and hence D G cross DG convolutions over all so the number of multiplications in the convolution of one kernel over the entire input f is DG square times D K square times M now this is for just one kernel but if we have n such kernels which makes the absolute total number of multiplications become n times D G square times D K square times M multiplications let's now take a look at


depth wise separable convolutions in standard convolution the application of filters across all input channels and the combination of these values are done in a single step def y separable convolution on the other hand breaks us down into two parts the first is depth wise convolution that is it performs the filtering stage and then point wise convolution which performs the combining stage let's get into some details here depth wise convolution applies convolution to a single input channel at a time this is different from the


standard convolution that applies convolution to all channels let us take the same input volume F to understand this process F has a shape D F cross D F cross M where D F is the width and height of the input volume and M is the number of input channels like I mentioned before for depth wise convolution we use filters or kernels K of shape DK cross DK cross one here DK is the width and height of the square kernel and it has a depth of 1 because this convolution is only applied to a channel unlike standard convolution


which is applied throughout the entire day and since we apply one kernel to a single input channel we require M such DK cross DK cross one kernels over the entire input volume F for each of these M convolutions we end up with an output DG cross DG cross one in shape now stacking these outputs together we have an output volume of G which is of shape DG cross DG cross M this is the end of the first phase that is the end of depth wise convolution now this is succeeded by point wise convolution point wise convolution involves performing the


linear combination of each of these layers here the input is the volume of shape DG cross DG cross M the filter K PC has a shape one cross one cross M this is basically a 1 Cross 1 convolution operation over all M layers the output will thus have the same input width and height as the input D G cross DG for each filter assuming that we want to use some n such filters the output volume becomes D G cross DG cross n so that's great we got this down now let's take a look at the complexity of this convolution we can split this into two


parts as we have two phases first we compute the number of multiplications in depth wise convolution so here the kernels have a shape DK cross D K cross 1 so the number of multiplications on one convolution operation is all DK times DK DK square when applied over the entire input channel this convolution is performed DG x DG number of times so the number of multiplications for the kernel over the input channel becomes DG square times DK square now such multiplications are applied over all em input channels for each


channel we have a different kernel and hence the total number of multiplications in the first phase that is depth wise convolution is M times D G square times D K square next we compute the number of multiplications in the second phase that is point wise convolution here the kernels have a shape one cross one cross M where m is the depth of the input volume and hence the number of multiplications for one instance of convolution is M this is applied to the entire output of the first phase which has a width and height


of D G so the total number of multiplications for this kernel is d G times D G times M so for some n kernels will have n times D G times D G times M such multiplications and thus the total number of multiplications is the sum of multiplications in the depth wise convolution stage plus the number of multiplications in the point-wise convolution stage we can take M times D G squared common now we compare the standard convolution with depth wise convolution we get the ratio as the sum of reciprocal of the depth of output


volume that is n and the reciprocal of the squared dimensions of the kernel DK to put this into perspective of how effective depth wise convolution is let us take an example so consider the output feature volume n of 1024 and a kernel of size 3 that's DK is equal to 3 plugging these values into the relation we get zero point 1 1 2 in other words standard convolution has 9 times more the number of multiplications as that of depth Y separable convolution this is a lot of computing power we can also quickly compare the number of parameters


in both convolutions in standard convolution each kernel has k times D K times M learn about parameters since there are n such kernels there are n times M times D K squared parameters in depth by separable convolutions will split this once again into two parts in the depth wise convolution phase we use M kernels of shape DK cross DK in point wise convolution we use n kernels of shape 1 Cross 1 cross M so the total is M times DK square plus M times n or we can just take M common taking the ratio we get the same ratio as we did for


computational power required so we understood exactly what depth wise convolution is and also its computation power with respect to the traditional standard convolution but where exactly has this been used well there are some very interesting papers here the first is on multi model neural networks these are networks designed to solve multiple problems using a single network a multi model network has four parts the first is modality Nets to convert different input types to a universal internal representation then we have an encoder


to process inputs we have a mixer to encode inputs with previous outputs and we have a decoder to generate outputs a fundamental component of each of these parts is depth wise separable convolution it works effectively in such large networks next up we have exception a convolution neural network architecture based entirely on depth wise separable convolution layers it has shown the state-of-the-art performance on large datasets like Google's jft image data set it's a repository of 350 million images with 17,000 class labels


to put this into perspective the popular image net took 3 days to Train however to Train even a subset of this jft data set it took a month and it didn't even converge in fact it would have approximately taken about three months to converge how'd they let it run to its full length so that's useful this paper is pushing convolution neural networks to use depth Y separable convolution as the de facto up third we have mobile Nets a neural network architecture that strives to minimize latency of smaller scale networks so


that computer vision applications run well on mobile devices mobile nets used F Y separable convolutions in its 28 layer architecture this paper compares the performance of mobile nets with fully connected layers versus depth wise separable convolution layers it turns out the accuracy on image net only drops a 1% while using significantly less number of parameters from twenty nine point three million the number of parameters it's down to just 4.2 million we can see the mulch as the number of multiplications and additions which is a


direct measure of computation has also significantly decreased for depth by separable convolution mobile Nets so here are some things to remember in this video depth Y separable convolution decreases the computation and number of parameters when compared to standard convolution second is that depth Y separable convolution is a combination of depth wise convolution followed by a point wise convolution depth wise convolution is the filtering step and point wise convolution can be thought of as the combination step


finally they have been successfully implemented in neural network architectures like multi model networks exception and mobile nets and that's all I have for you now thank you all for stopping by today if you liked the video hit that like button if you want to stick around hit that subscribe button if you really want to stick around hit that Bell icon next to the subscribe button so as to be notified of my uploads immediately links to important papers are down below so check them out have a good day and I'll see you in the


i-it's it's a wonderful to be here I have been remiss in that I have not been to Prague for for a decade so it was it's wonderful to be back in Prague and it's wonderful to be in this fancy new Institute and so I think because because there is various types of people here there are some vision people some graphics people and some some others that learning done learning I'm going to give you kind of a an overview of some of the stuff that we have been doing but don't go into too much detail because that might be too boring for others and

Alexei Efros: Self-supervision, Meta-supervision, Curiosity: Making Computers Study Harder


and and maybe too much for a foursome but then I'm I'm here the rest of the day so I'm happy to to chat about these things more so of course you know how it is with with with with with professors you know all the work has done by the graduate students and the postdocs who are amazing and then the professor just pushed together in the slides and and and and presents the work and in this case actually even the slides most of them have been done by their by the Graduate since so I'm really just just a audio recording of really the all the


wonderful work that that they have been doing so probably if you didn't live under a rock for the last few years you have heard of the deep networks and how they have revolutionised computer vision and kind of the standard classic way of doing this is it's basically a classic supervised learning problem you are giving a network which you can think of as a big black box a pairs of input images and output labels XY pairs okay and this big black box essentially you can think of it as memorizing these these the score currents or the the its


memory its modeling the associations between the XS and the Y's okay and of course you need to have lot and lots and lots of these training pairs so you have lots of people clicking on a bunch of imager a lots of you know millions of images and what do they what is their labels and once you have trained on millions of these pairs of images and labels then given a new image this magic black box can tell you what label it is and this is what supervised direct supervised learning and this particularly deep learning has


been all about ok but there are some problems that that I'll mention of this beautiful story one obvious problem is that this this labeling bit is very expensive millions of images don't come cheap you have to have people actually label them and for every new problem we need to label more images so that's that's that's a problem of cost but there is also another problem that is a little bit more subtle and here is an illustration of this so here is an image that is basically you can think of it as a texture synthesized version of the


original image you basically some of the pixels have been moved around in certain ways but to kind of to preserve the kind of the statistics of that of the image it's it's it's actually work by by Leon Gattis ok so we change the input but the neural network is perfectly happy to still call it a collie in fact I can give you other random images like this right and it's still basically happy to call it by that same class collie ok and what this suggests is that this magical black box the convolutional neural network is not actually doing that much


it's not doing what we think of a lot of computer vision should be all about it's not doing you know figure-ground detection it's not fine the the you know the the foreground region it's not finding the dog it's not segmenting the dog out from the background it's not doing any of the foreground background or occlusion reasoning none of that it doesn't need to do any of that because it probably just looks at the snout in a couple of eyes and then says oh yeah that's a collie okay and maybe some his called histogram so it doesn't need to work


super hard to solve this problem okay and this is you know a cause for worry because in this particular case this is imagenet classification task so you have a thousand classes and so maybe this is not maybe you don't need to work that hard because you might not need to have to to to really worry about oops you might not need to worry about finding the boundaries of objects so even reasoning about objects with only a thousand classes but the issue is that we don't really have more than a thousand classes labeled and so in the


end what this magical neural network is doing it's not really object detection it's really more like texture classification it's classifying dogcatcher collie texture so this if you know 1,000 weight classification tasks is the only thing you want to do maybe this is not so bad but here is an example of something that you know Joseph and and and a lot of us have been worried about for a long time action recognition okay action recognition the same thing but in time so you you have you have a video and you want to


recognize what action is being performed and the one very weird result in action classification has been that giving more frames of video to the classifier did not seem to improve performance that just a single frame oh thank you perfect the single flame is is good enough all right look at that perfect okay thank you okay so for example so you basically for a single frame you do basically just as well and this was a big strange result that people don't know why it was so but if we look at for example here is an


opening fridge action so you go to the fridge you extend your hand you pull on the fridge door you open the fridge and then you close it right and you want to recognize other actions that are opening fridge actions okay if you run a classifier for this task you label a whole bunch of opening the fridge actions as positives and then others as negatives you train your network and then if you look at what the performance is great by the way the performance is very very good but if you look at which frames did the classifier actually pay


attention to it's just this one so it doesn't care it doesn't track the hand it doesn't care about the the fridge door really all it cares about is again it's the texture of an open refrigerator okay and once it sees an open refrigerator texture it knows oh this must be a fridge opening action what else could it be right so again it's it's taking the easy way out it's being lazy because it doesn't have to work hard okay and maybe if you asked me a year or two ago you know how do we deal with this problem I would say that


this is all an issue of a data set bias that the only pictures of opening refrigerators are in this opening fridge action so let's add some negative images of just open fridges like from from you know maybe Amazon product search and then everything will be fine now I'm starting to think that while data set bias is a problem it's not the whole problem because in a sense this data set bias will never go away it's there is no way that the data the data is finite so they will never be able to fix all the holes there will


always be a way to cheat because if there is a finite amount of data there is always a way to to find a path that that is that is somehow you know cheating through the data and and so it's it's there is it's kind of like playing a you know this game children's game of Wacom all you you you you push something down and something else pops up okay and and also if you think if you ask a machine learning people about it this this is not even their problem because the machine learning people say look you train on the training set and


then you've evaluated on the validation set over your method or over your data set okay so you take your data set you split it into two the training and the test set and as long as it does well on the test set you're fine right and the test set comes from the same distribution as the training set so it's the same statistics what we have here is that we want to test our algorithms on something that's not really in the in the test set of the of the data set that's something else so we train on say detecting cars from from imagenet data


set but then we want to go out on the street and and and detect cars there and the cars on the street don't really have the same distribution as the cars that you were trained on but we still want to do it so in a sense our problem is actually quite somewhat different than the problem with in machine learning that we actually do want to test on things we never really trained on so we wanted to really be general and so the way forward I see is that somehow we need to better use the data we have there is no hope to ever get all the


data that we that that that that will make the problem perfectly can concrete but we need to somehow use the data that we have better I'll give you a couple of examples of the way I think about this so in in a you can think of it as as a way that a well around country is run compared to a badly run country's run it's not that in a well-run country you cannot cheat the laws of course you can but the system is set up in such a way that it's actually more expensive to cheat than to follow the law okay and so even though you can cheat you don't do


it because it's not in your self-interest in a poorly run country it's cheaper to cheat so everybody cheats and there is no way you know there is nothing you can do about it okay so we need to somehow set up our problem is such a way that it's more expensive for the network to cheat let the easiest thing the network can do is to do the right thing that's that's the goal okay and if you think about the way that we do this direct supervised learning input image output label and just train on these pairs that just sets up sets up


your your life for cheating because so in in in in in my class you know we have a whole semester worth of material and then the debt of the semester there is a final exam so of course most people don't do anything during the whole semester the night before the exam they they look at some exams from previous years and they try to memorize this you know the question and answer question answer question answer question alright and they basically memorize the whole all of this set of question and answer pairs and then they go to the examine


actually they do pretty well right this is not a bad strategy to pass the final exam it's a bad strategy if you actually want to learn but it's not the best strategy to pass the exam because you know I'm lazy I'm going to make the exam this year to be not that different from the exam last year in any case there is very small set of problems you can ask that is easy to grade etc etcetera right and so this kind of memorization of question answer question answer it's actually the correct thing to do if your goal is to


pass the exam but of course our goal is not to pass the exam our goal is to to actually learn the material so how do you learn the material how do you actually learn you know arithmetic for example when you're a little kid to really learn it what you do is you don't get yourself question-answer pairs you look at the question you try really hard to solve it and then once you solve it you got some answer you go to the back of the book to compare it with them with the right answer and then that's how you kind of try to update yourself okay and