GitHub Copilot may be perfect for cheating CompSci programming exercises - Page 1
Forum section: electronics / general

author: xrunner
author_rank: Super Contributor
timestamp: August 20, 2022, 11:07:41 am
content: Quote

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: August 20, 2022, 12:47:24 pm
content: My question is: what good does the cheating get you when you're dropped into the workforce and you know, have to solve actual problems?Sound's like a naive question certainly but if you could expend that much effort whist exposing yourself to risk getting caught, wouldn't you just study the module?

author: bd139
author_rank: Super Contributor
timestamp: August 20, 2022, 02:38:09 pm
content: Had some problems with obvious cheating on entry tests before. Thus we give people a computer without a network connection when we test themAlso one question is impossible to answer. And it’s early on. Anyone who knows their ass from their elbow would spend 2 minutes realising that. Anyone who doesn’t burns the rest of the test scratching their head.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 20, 2022, 07:03:43 pm
content: Quote from: Ed.Kloonk on August 20, 2022, 12:47:24 pmThat's a good question, but it ignores a fact: a non-negligible fraction of people pursuing an engineering degree of some kind just get it for the social status and the selection advantage they get for some less-technical jobs, such as management, sales, marketing in some areas, etc. So ultimately they don't care whether they are capable of applying what they are supposed to have learned and mastered - they just want the degree for the benefits it brings them.

author: RoGeorge
author_rank: Super Contributor
timestamp: August 20, 2022, 08:40:25 pm
content: Ads are getting smarter. Best add for Microsoft's Copilot to get student's undivided attention, "can help you cheat the exams".(though Copilot is now a payed product, the free GitHub Copilot beta ended this year)OTOH, yes, transformers will be a game changer in many fields (not only programming).Not to say, knowing the fundamentals of programming will become as relevant as it is for electronics to learn how to operate a sliding ruler in the age of LTspice simulators.From now on, the programming as we know it will change dramatically. Only days ago seen what a model of 480MB can do, all running locally on an i7 desktop from 5+ years ago.It's already creepy what transformers based AI can do (GPT stands for Generative Pre-trained Transformer). Microsoft's Copilot is based on GPT3, just like the one I was fiddling with recently (sorry for the shameless plug, but it shows examples of generated text and code, running on a casual desktop and with small generic models, all running offline and trivial to setuphttps://www.eevblog.com/forum/programming/ai-hello-world/).With better hardware and bigger, more specialized models running in the cloud, programming as we know it will go extinct in 5 to 10 years most.GPT AI is a revolutionary technology. CodeBERT model (480 MB, 125 mil parameters) can understand plain English requests and generate code that fits the description, can do that in 9 programming languages, and faster that it would take one to type all that.Code helpers will revolutionize programming. You won't believe until you'll try for yourself something like CodeBERT or GPT-Neo (which are free and open source, zero AI skills required to install or run them). Give it a try.

author: AndyBeez
author_rank: Frequent Contributor
timestamp: August 20, 2022, 08:45:14 pm
content: Quote from: bd139 on August 20, 2022, 02:38:09 pmNo network, but how can they access Instragram in the middle of the test? You are SO not going to attract the most talented candidates doing that.In fairness, using Copilot is a form of initiative, showing use of the tools available. Anyway, Copilot understands the questions. Questions that are often designed by academics who've never programmed anything off a white board. "Implement a snapshot array supporting pre-defined datatypes"Sometimes questions are just too stupid in their scope. For example, "how would you redesign Google's database for web indexing?" Answer is, "f*** knows, ask Alexa." Or, "tell me how you used data to measure impact?" Response, "f*** knows, they all died."

author: bd139
author_rank: Super Contributor
timestamp: August 20, 2022, 08:47:55 pm
content: Hahaha.Problem we have is that we have source escrow with some customers and strict security requirements. So Copilot is actually an export risk as it produces copyright code and insecure shit galore.There is nothing good about it at all. It's dangerous.

author: RoGeorge
author_rank: Super Contributor
timestamp: August 20, 2022, 09:16:24 pm
content: - Insecure? As in, the same with any existing software?- About license, maybe Microsoft Copilot doesn't show license, never tried it, but other helpers can specify the license. For example this one, Codeon, a free programming helper that tells the license, too:Another thing is that GPT can do more than just a search retrieval, it can create new lines, code that was never shown to it during training.

author: bd139
author_rank: Super Contributor
timestamp: August 20, 2022, 09:19:24 pm
content: Insecure as in it makes people lazy enough to not think of security concerns and review properly.As for license our lawyers said we can't and we don't argue with themI am really not interested in any solution around this personally. It's absolutely not the way we should be writing software in any way shape or form.

author: RoGeorge
author_rank: Super Contributor
timestamp: August 20, 2022, 09:52:56 pm
content: Yes, with time this can make programmers "weaker", it makes one less self-reliant, and more dependent on 3rd factors.Though, this is the same as any other technology does, or any other helper. Writing weakened memory, horses weakened running, and so on. Any advance in comfort will make us less reliant, and in time we become addicted to that comfort. Take electricity for example. Civilization will crumble in a matter of weeks without electricity.It's been a long time since we were able to procure our food from wilderness, and grow our own fur to stay warm.

author: janoc
author_rank: Super Contributor
timestamp: August 20, 2022, 11:39:47 pm
content: Maybe a heretical idea - but perhaps Copilot is the best thing that could happen for CS education (and possibly also software developer interviews).It is about time these idiotic assignments were put out to pasture. They don't test anything meaningful, only that the person taking the test can type "palindrome" or "fizzbuzz" into Google and copy & paste stuff from StackOverflow. Copilot only made it easier (with a bonus free plagiarism/copyright infringement ...)Try to give students an assignment that is new (and thus cannot be simply found online), interesting and that (gasp!) requires then to actually think about the problem. And, crucially, after the test ask them to explain their solution! 99% of cheaters have no clue whatsoever about what they have copied.Yes, it is more work for the teacher than just giving the same quiz questions year after year for the last 20 years. But assessment also doesn't need to be only quizzes!For background: I have taught C, C++, Python & OpenGL programming to university students in the past. Been there, done that. My university teaching portfolio was based on how to teach and assess these things, without going crazy with inventing assignments and having to mark all of then myself (no TAs to offload the task on!) and without having to deal with rampant cheating.The best way to deal with cheating is to embrace it - students will cheat, period. Threats, honor codes, etc. don't really work. So turn the tables - allow them to cheat! But then give them assignments where the cheating won't help them any because the assignment can't be copied from a place online (or a book). And if they still do cheat (e.g. by cribbing it from a classmate), it is an extra learning opportunity for them.What is important is whether they actually understood the subject - andthatis what gets assessed, not whether or not they memorized some algorithm or copied that FizzBuzz from StackOverflow. I was happy to give them an A as long as they could demonstrate they understood the topic in the exam - and that wasn't done by submitting a test but by an oral exam**. I couldn't care less whether they learned it from the lecture, book, code copied online or copied from a classmate.**That could obviously be problem in large classes but then it is a matter of organization, spot checking the tests by asking questions, doing the orals by groups and asking questions there, even replacing the written exam by a term long larger project to be made and turned in (perhaps as an alternative to the test/exam), etc. Many solutions are possible as long as the teacher has open mind beyond the lecture -> written test tradition.

author: magic
author_rank: Super Contributor
timestamp: August 21, 2022, 06:37:33 am
content: Quote from: bd139 on August 20, 2022, 02:38:09 pmVery evil, I like itQuote from: evb149 on August 21, 2022, 12:04:18 amThat's true, but AI is not the solution because of its statistical nature and the resulting inevitable degree of error and unreliability. It sounds cool to describe what you want in human language and have the machine generate the boilerplate, until you realize that sometimes even asking competent human programmers in human language results in code slightly different than you wanted.That's a job for IDE plugins or custom scripts, which need to ingest a formalized and unambiguous description of what's to be generated and transform it with ordinary deterministic algorithms. If the output is wrong despite your specification being correct, that's a bug and it can be traced down to an exact line of code in the tool and fixed. Have fun doing that with AI. Such things also tend to run faster than AI for some reason.IMHO a big reason why there is lots of ugly boilerplate code difficult to maintain in the wild is largely due to programmers being too lazy to automate what already can be automated with tools we have had for decades. And because many widespread languages aren't making the task particularly easy, by being a nightmare to parse. Trying to solve this with AI will only make things worse. I envision a future where several multi-gigabyte revisions of some AI model are included in a few megabyte source repository to generate a few files that can only be generated by a particular version of a particular AI that some developer employed at a particular time. We will call it "AI hell" and long discussions will be had on "H".N. how to deal with it using shared AI repositories, AI as a service, and private AI cloud.

author: RoGeorge
author_rank: Super Contributor
timestamp: August 21, 2022, 07:37:12 am
content: If AI write bugs, then "use the stones to destroy the stones".Research results are encouraging already:https://2021.msrconf.org/details/msr-2021-mining-challenge/3/Applying-CodeBERT-for-Automated-Program-Repair-of-Java-Simple-BugsThat won't replace all the CS gray-beard in a year, but I think in 10 years AI generated code will be the norm.We all need to understand thatthe working principles of AI are no different from how the human mind is working, therefore AI would be able to do everything a human can, in terms of intelligence. It is only a matter of the size of the network that still gives us an advantage. The soon we recognize we are not the center of the Universe, the better.We are all in denial right now, but there is knowledge in piles of data, and we just learn how to make use of that.Any pile of data also stores knowledge, and that can be extracted by simply classifying the data, by looking at similarities and correlations. It doesn't matter what that data is about, it can be literature/video/audio/programming/whatever. And once you have the knowledge, the patterns, the notions learned from data, you can use the patterns to extrapolate, and thus create something new, or simulate a hypothesis in order to predict what will happen, and so on.That is why having a brain is advantageous, because it can predict the outcome in advance. You don't have to wait for the tiger to kill you first, you can "see" the outcome, you can predict, and plan for how to escape fate. Otherwise why keeping a brain, it's a very energy intensive organ and hard to feed.The brain is an Oracle, it can predict the near future. We wouldn't be able to do the simplest thing, like walking a room, without predicting the outcome of each and every step in advance. This is happening in an automated way, and we are not aware of it. We think we "see" the surrounding, but experiments tells we rather simulate the surrounding. The data stream coming from the eyes is only adjusting the simulation so it won't diverge too far. The brain is continuously simulate what we "see" or "feel"in advance.Mind is nothing but a driven illusion.A very brief simulation, in regards to the complexity of the objective reality that surrounds us.Back to our very recent AI breakthroughs, I think the true value of collecting and hoarding data is exactly this, the unseen knowledge stored in that data, targeted commercial is peanuts. Data rush is the new gold rush. Whoever will get that faster will become more powerful. A human mind can not process billions of records, machines can. That will reveal patterns no human can see.Yes, AI it's statistical and can make mistakes. Just like us, humans. We both have these pitfalls because we work on the same principles. We extract knowledge from floods of data streams. We compress past experiences into nuggets of wisdom. This is what the AI has achieved, too. It is happening, denial won't make it vanish.There is one last straw that we still have, while the machines don't have it yet, self-reproduction.

author: bd139
author_rank: Super Contributor
timestamp: August 21, 2022, 08:03:48 am
content: Some pretty big assumptions there.Let’s not use AI as a term here as that is a bridge too far here. ML applies a corpus of things humans did previously to new problems. It does so with the skill and altitude of a small, retarded cat. And most of the corpus is written by slop sucking morons. The problem is that it’s really important to understand how A turned into B and that’s pretty much impossible to determine unless you store the entire model and state with the decision which is impractical. Really with all processes it’s garage in, garbage out.This is ultimately a dead end. I have watched two entire companies burn trying to get ML driven modelling for risk calculation working anywhere near as reliably as simple rules. In one case the model actually decided to approve a mortgage to an Iranian terrorist. And that’s exactly the same situation we’re in with code. Simple rules are better. By a mile. They are reliable, well understood, consistent and have consensus built into them. Which is exactly what businesses need.We should invest in building tools that make it harder to make mistakes not tools which purportedly allow the scope of risk to be increased.

author: magic
author_rank: Super Contributor
timestamp: August 21, 2022, 08:25:09 am
content: I absolutely agree that it will be used, as you have already witnessed yourself, and denial won't make it vanish.

author: bd139
author_rank: Super Contributor
timestamp: August 21, 2022, 08:44:00 am
content: You’re right. I see it is another risk we have to defend against. That’s hard enough already when you hire the lowest biddersI’m counting down until i retire at this point

author: magic
author_rank: Super Contributor
timestamp: August 21, 2022, 09:01:51 am
content: Too young for that. I'm waiting for things to deteriorate so badly that it becomes socially acceptable to just smash the head of anyone you dislike with no consequences, as in the good old times

author: Towger
author_rank: Super Contributor
timestamp: August 21, 2022, 09:06:12 am
content: In my day, you sat in an exam hall with paper and pen and wrote the code out by hand. If you went to the lectures / did all the assignments (your self) the questions were relatively easy to answer. Those that did not put in the work were screwed. This old school system did require the examiners/lectures time and effort to read through and correct, but there was no 'cheating' with online resources. Not that there was much back then...

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 21, 2022, 10:56:09 am
content: Quote from: bd139 on August 20, 2022, 02:38:09 pmI would, most definitely: I'm addicted to seemingly impossible programming problems. But I do claim I know my ass from my elbow (as I got two elbows, and they have no cheeks, but one ass, that has two big, meaty, hairy cheeks).Also, I always keep a terminal or browser window open to the man pages or language documentation when programming. I don't waste my feeble brain power in memorizing details like the parameter order in API calls; I don't always even remember the exact function call names. (Of course, in Linux, one can usually find these already installed in/usr/share/doc/package/, so no network connection is needed.)I have been ridiculed before because of that, although the ridicule dies down in a couple of weeks or months, when the quality of my work product is observed. Then, somehow, the same practice seems to become the norm among my colleagues, too. Go figure.All kidding aside, what I absolutely detest in programming questions is when the obvious/sane/normal/preferred solutions are excluded by insane requirements. Like "Write a Hello World program with no more than three 'i's and two 'e's in your source code."

author: bd139
author_rank: Super Contributor
timestamp: August 21, 2022, 11:05:37 am
content: It's a good trap. We have a big problem with people not being solution focused and will blow weeks if not months on dead ends and trendy shit rather than solve the problem.Actually the best impossible situations are the infra ones. I love to throw out the question:So AWS eu-west-1 is completely down and has been down for 2 days due to a fuck up at Amazon. Also there is no capacity in eu-west-2 because anyone who could moved there did so and used it all up. We can't move to another region due to compliance constraints. Customers are in SLA violation now. What do we do?I hired the guy who said "hit cwjobs"

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 21, 2022, 11:58:35 am
content: Quote from: bd139 on August 21, 2022, 11:05:37 amThat's true, even I can't argue with that!Quote from: bd139 on August 21, 2022, 11:05:37 amYup. "Be fucked, and try to avoid getting fucked by a similar situation in the future" would be my answer now.I have been in a somewhat similar situation in my student days, when I had a summer coding project from early May to end of August (a simple scheduling web widget), but had to deal with stepwise delays in the infrastructure up till mid-August. My fault for not stopping the shit show in June, though; I'd never had such a drip-drip of issues before, and honestly thought I could fix stuff in time. In all of my career, I've failed fewer projects than I have fingers in one hand. Still annoys me to this day, and would love to apologise and explain to those who suffered because of the failure of that project.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 21, 2022, 06:10:44 pm
content: Quote from: bd139 on August 21, 2022, 08:03:48 amYup. Which is the basis of what humans have been doing all along. Except that we humans have *some* ability to do this with a bit of discernment, while ML, in its essence, fundamentally *can't* have discernment. We also have this fun tendency to take pride in finding new ways of achieving things - that's called creativity or something, along with some ego - and that's actually very useful. ML is not going to achieve this.But ultimately, that's just a bunch of people trying to push ML as hard as they can, not because this is actually beneficial to anyone here, but because they have a vested interest in it. There's huge money to be made regardless of the outcome. This is just a huge cash cow.

author: bd139
author_rank: Super Contributor
timestamp: August 21, 2022, 06:12:17 pm
content: Nailed it. Especially the last comment.

author: AndyBeez
author_rank: Frequent Contributor
timestamp: August 21, 2022, 10:20:30 pm
content: Is it ML or MC? That's machine copying. It makes one wonder what very wrong answer on stackoverflow from 10 years ago will be inducted into a code solution flying over the ocean? And who or what will be accountable for the memory leak that opened the aircraft doors in-flight? That's a solution for the AI lawyer.Quote from: bd139 on August 21, 2022, 11:05:37 amThat's Kobayashi Maru :https://en.m.wikipedia.org/wiki/Kobayashi_Maru

author: bd139
author_rank: Super Contributor
timestamp: August 21, 2022, 10:29:07 pm
content: Kobayashi Maru was the inspirationAs for the AI lawyer issue, that's another problem. Google are already using AI police:https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 21, 2022, 10:57:53 pm
content: Is that the world we really want to live in?Just asking. Because if the answer is no for a majority of us, gotta ask who is driving it.

author: magic
author_rank: Super Contributor
timestamp: August 22, 2022, 06:07:57 am
content: Quote from: SiliconWizard on August 21, 2022, 10:57:53 pmJust think of the children, dude.Majority of us aren't pedophiles so yes, this is the world we want to live in

author: AndyBeez
author_rank: Frequent Contributor
timestamp: August 22, 2022, 08:48:32 am
content: I ponder how Google AI learns what is an indecent image, is their a CAPTCHA for this? I am a  human. Select all images with butts.Quote from: SiliconWizard on August 21, 2022, 10:57:53 pmIt's being driven by our tech ignorant political class and the silicon valley sociopaths with snake oil as a solution SOaaS to sell - for a lot of tax payers dollar$. Tax payers who are brainwashed into believing they need protecting from the evil that is the internet, via their congressman's Twitter account.There is a very serious cyber security issue here: If Google on behalf of the NSA are using AI to scrape the content of our emails to "protect children", and all with our assumed consent, what else is being examined to protect us from "perverts, terrorists and communists"? How much business to business traffic sent over SSL is being 'identified' to be examined by a human analyst?Simply, do not use GMail as it is a security risk. Use your own or trusted email server, or encrypt sensitive contents. It's what the real bad actors have been doing for decades.

author: janoc
author_rank: Super Contributor
timestamp: August 22, 2022, 06:26:43 pm
content: Quote from: AndyBeez on August 22, 2022, 08:48:32 amAnd it is a sure fire way to get your e-mail tagged as spam and rejected/trashed unless you are big corporation, passing your e-mail through a large ISP/vendor (and then you are back to square one) or willing to deal with a ton of hassle, aggravation and costs figuring out why your e-mails are not being delivered, getting lost, bounced, etc.That is in addition to spam, malware and constant security related stuff (updating the server, hardening it, warding off constant hacking attempts by bots, ...) which are the daily bread and butter for anyone running any kind of server on the internet.Sadly the time when anyone could run their own server without issues as long as they were a bit technically competent are long gone.Also if you start encrypting, that's only going to attract even more attention to you because you are standing out like that proverbial guy in a gorilla suit in the middle of party crowd - "Hey look - that guy has something to hide!"Doesn't mean you shouldn't do these things but it is far from a clear-cut and obvious solution. The tradeoffs are fairly significant there.